[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:13:11 dist-test-slave-hllj krb5kdc[29289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:13:11 dist-test-slave-hllj krb5kdc[29289](info): set up 2 sockets
Sep 26 02:13:11 dist-test-slave-hllj krb5kdc[29289](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 02:13:12.702643 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:13:12 dist-test-slave-hllj krb5kdc[29289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:13:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:13:12.873204 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:40115
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:40115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:13.191548 29302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:13.191843 29302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:13.191953 29302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:13.199330 29302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:13.199445 29302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:13.199494 29302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:13.199571 29302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:13.199621 29302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:13:13.210035 29302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:40115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:40115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:13.211373 29302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:13.212708 29302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:13.224938 29307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:13.224882 29310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:13.224965 29308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:13.225201 29302 server_base.cc:1034] running on GCE node
I20260926 02:13:13.226344 29302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:13.227883 29302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:13.229089 29302 hybrid_clock.cc:648] HybridClock initialized: now 1790388793229039 us; error 40 us; skew 500 ppm
I20260926 02:13:13.229462 29302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:13 dist-test-slave-hllj krb5kdc[29289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388793, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:13.232425 29302 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:13:13.233886 29302 webserver.cc:460] Webserver started at http://127.28.152.190:35523/ using document root <none> and password file <none>
I20260926 02:13:13.234539 29302 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:13.234635 29302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:13.234879 29302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:13.237874 29302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "60f695cc4be24769b17b2bd4acd58cdb"
format_stamp: "Formatted at 2026-09-26 02:13:13 on dist-test-slave-hllj"
server_key: "a0134a04a30b7fb73b61bbed7ca4d488"
server_key_iv: "82694e07c5eb401128008b00dd062dde"
server_key_version: "encryptionkey@0"
I20260926 02:13:13.238567 29302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "60f695cc4be24769b17b2bd4acd58cdb"
format_stamp: "Formatted at 2026-09-26 02:13:13 on dist-test-slave-hllj"
server_key: "a0134a04a30b7fb73b61bbed7ca4d488"
server_key_iv: "82694e07c5eb401128008b00dd062dde"
server_key_version: "encryptionkey@0"
I20260926 02:13:13.243697 29302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:13:13.247248 29317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:13.248661 29302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:13:13.248811 29302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "60f695cc4be24769b17b2bd4acd58cdb"
format_stamp: "Formatted at 2026-09-26 02:13:13 on dist-test-slave-hllj"
server_key: "a0134a04a30b7fb73b61bbed7ca4d488"
server_key_iv: "82694e07c5eb401128008b00dd062dde"
server_key_version: "encryptionkey@0"
I20260926 02:13:13.248984 29302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:13.271462 29302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:13.275640 29302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:13.275895 29302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:13.293745 29302 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:40115
I20260926 02:13:13.293769 29368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:40115 every 8 connection(s)
I20260926 02:13:13.295109 29302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:13.298817 29369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:13.299916 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 29302
I20260926 02:13:13.300220 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:13.302645 29282 external_mini_cluster.cc:1428] Setting key 8a39602e8921559d114b91c7568efea2
I20260926 02:13:13.311376 29369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb: Bootstrap starting.
I20260926 02:13:13.314374 29369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:13.315382 29369 log.cc:824] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:13.318996 29369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb: No bootstrap required, opened a new log
I20260926 02:13:13.327630 29369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f695cc4be24769b17b2bd4acd58cdb" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40115 } }
I20260926 02:13:13.328150 29369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:13.328263 29369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f695cc4be24769b17b2bd4acd58cdb, State: Initialized, Role: FOLLOWER
I20260926 02:13:13.328912 29369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f695cc4be24769b17b2bd4acd58cdb" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40115 } }
I20260926 02:13:13.329123 29369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:13.329278 29369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:13.329452 29369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:13.331764 29369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f695cc4be24769b17b2bd4acd58cdb" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40115 } }
I20260926 02:13:13.332278 29369 leader_election.cc:302] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60f695cc4be24769b17b2bd4acd58cdb; no voters:
I20260926 02:13:13.332823 29369 leader_election.cc:288] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:13.332975 29374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [term 1 FOLLOWER]: Leader election won for term 1
Sep 26 02:13:13 dist-test-slave-hllj krb5kdc[29289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:13.334038 29374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [term 1 LEADER]: Becoming Leader. State: Replica: 60f695cc4be24769b17b2bd4acd58cdb, State: Running, Role: LEADER
I20260926 02:13:13.334646 29374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f695cc4be24769b17b2bd4acd58cdb" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40115 } }
I20260926 02:13:13.335011 29369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:13.338032 29376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60f695cc4be24769b17b2bd4acd58cdb. Latest consensus state: current_term: 1 leader_uuid: "60f695cc4be24769b17b2bd4acd58cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f695cc4be24769b17b2bd4acd58cdb" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40115 } } }
I20260926 02:13:13.338414 29376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:13.338639 29375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60f695cc4be24769b17b2bd4acd58cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f695cc4be24769b17b2bd4acd58cdb" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40115 } } }
I20260926 02:13:13.338891 29375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:13.340634 29380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:13.347781 29371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:13.307369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.28.152.190:40115)
0926 02:13:13.308266 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:13.308286 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:13.309598 (+ 1312us) server_negotiation.cc:407] Connection header received
0926 02:13:13.319248 (+ 9650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:13.319277 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:13.319790 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:13.320235 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:13.323274 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:13.325383 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:13.327156 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:13.328212 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:13.328420 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:13.334974 (+ 6554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:13.334986 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:13.335020 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:13.335079 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:13.339551 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:13.340692 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:13.340703 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:13.340714 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:13.340970 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:13.341648 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:13.341653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:13.341656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:13.342394 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:13.342715 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:13.345636 (+ 2921us) server_negotiation.cc:299] Negotiation successful
0926 02:13:13.346178 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":400,"threads_started":1}
I20260926 02:13:13.346699 29380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:13.351197 29391 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:13:13.357561 29380 catalog_manager.cc:1347] Generated new cluster ID: 7ebedd4da1774b4d94fb367cecfd8325
I20260926 02:13:13.357694 29380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:13.382863 29380 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:13.384433 29380 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:13.392688 29380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60f695cc4be24769b17b2bd4acd58cdb: Generated new TSK 0
I20260926 02:13:13.393590 29380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:13:13.445345 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:40115
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:13.799794 29395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:13.800107 29395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:13.800302 29395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:13.808054 29395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:13.808197 29395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:13.808360 29395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:13:13.818528 29395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:40115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:13.820048 29395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:13.821537 29395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:13.835718 29403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:13.835784 29400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:13.835752 29401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:13.836606 29395 server_base.cc:1034] running on GCE node
I20260926 02:13:13.837333 29395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:13.838344 29395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:13.839538 29395 hybrid_clock.cc:648] HybridClock initialized: now 1790388793839478 us; error 53 us; skew 500 ppm
I20260926 02:13:13.839871 29395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:13 dist-test-slave-hllj krb5kdc[29289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388793, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:13.843104 29395 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:13:13.844630 29395 webserver.cc:460] Webserver started at http://127.28.152.129:43725/ using document root <none> and password file <none>
I20260926 02:13:13.845412 29395 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:13.845531 29395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:13.846269 29395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:13.849255 29395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "317e8cac93c54848b7e1596a49e850ae"
format_stamp: "Formatted at 2026-09-26 02:13:13 on dist-test-slave-hllj"
server_key: "3214b59232a0c509ca24c980935af31a"
server_key_iv: "c6f1d42491e05245b8b83eee47a7847f"
server_key_version: "encryptionkey@0"
I20260926 02:13:13.850033 29395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "317e8cac93c54848b7e1596a49e850ae"
format_stamp: "Formatted at 2026-09-26 02:13:13 on dist-test-slave-hllj"
server_key: "3214b59232a0c509ca24c980935af31a"
server_key_iv: "c6f1d42491e05245b8b83eee47a7847f"
server_key_version: "encryptionkey@0"
I20260926 02:13:13.855602 29395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 02:13:13.859221 29410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:13.860562 29395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:13:13.860718 29395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "317e8cac93c54848b7e1596a49e850ae"
format_stamp: "Formatted at 2026-09-26 02:13:13 on dist-test-slave-hllj"
server_key: "3214b59232a0c509ca24c980935af31a"
server_key_iv: "c6f1d42491e05245b8b83eee47a7847f"
server_key_version: "encryptionkey@0"
I20260926 02:13:13.860918 29395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:13.874369 29395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:13.878800 29395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:13.879087 29395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:13.880030 29395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:13.881413 29395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:13.881496 29395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:13.881610 29395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:13.881667 29395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:13.916059 29395 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:44547
I20260926 02:13:13.916092 29522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:44547 every 8 connection(s)
I20260926 02:13:13.917440 29395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:13:13.924116 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 29395
I20260926 02:13:13.924324 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:13:13.924889 29282 external_mini_cluster.cc:1428] Setting key 183e9fb8188aef23e00ee3aab970d930
Sep 26 02:13:13 dist-test-slave-hllj krb5kdc[29289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388793, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:13.938881 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:13.920177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:46307 (local address 127.28.152.190:40115)
0926 02:13:13.920631 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:13.920636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:13.921695 (+ 1059us) server_negotiation.cc:407] Connection header received
0926 02:13:13.923641 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:13.923646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:13.923747 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:13.923841 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:13.925626 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:13.926493 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:13.928278 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:13.928909 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:13.929053 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:13.932785 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:13.932790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:13.932793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:13.932830 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:13.935247 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:13.936226 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:13.936233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:13.936236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:13.936408 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:13.936865 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:13.936871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:13.936875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:13.937279 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:13.937441 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:13.938258 (+ 817us) server_negotiation.cc:299] Negotiation successful
0926 02:13:13.938500 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":233,"threads_started":1}
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
I20260926 02:13:13.939661 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:13.920780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:40115 (local address 127.28.152.129:46307)
0926 02:13:13.921528 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:13.921575 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:13.923332 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:13.924115 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:13.924123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:13.924718 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:13.925442 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:13.925463 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:13.926680 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:13.926686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:13.928105 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:13.928118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:13.929120 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:13.929151 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:13.929383 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:13.930050 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:13.930073 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:13.932525 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:13.935407 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:13.935418 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:13.935437 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:13.936028 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:13.936539 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:13.936542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:13.936545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:13.936743 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:13.937409 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:13.937416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:13.937840 (+ 424us) client_negotiation.cc:769] Sending connection context
0926 02:13:13.938090 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 02:13:13.938516 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":292,"threads_started":1}
I20260926 02:13:13.941272 29523 heartbeater.cc:347] Connected to a master server at 127.28.152.190:40115
I20260926 02:13:13.941622 29523 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
I20260926 02:13:13.942394 29523 heartbeater.cc:511] Master 127.28.152.190:40115 requested a full tablet report, sending...
I20260926 02:13:13.944622 29334 ts_manager.cc:194] Registered new tserver with Master: 317e8cac93c54848b7e1596a49e850ae (127.28.152.129:44547)
I20260926 02:13:13.946357 29334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:46307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:13:13.965660 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:40115
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:14.301090 29529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:14.301431 29529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:14.301565 29529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:14.308883 29529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:14.309024 29529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:14.309226 29529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:13:14.319476 29529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:40115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:14.320992 29529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:14.322722 29529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:14.337683 29535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:14.337606 29534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:14.337785 29537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:14.338348 29529 server_base.cc:1034] running on GCE node
I20260926 02:13:14.338997 29529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:14.340003 29529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:14.341231 29529 hybrid_clock.cc:648] HybridClock initialized: now 1790388794341167 us; error 57 us; skew 500 ppm
I20260926 02:13:14.341598 29529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:14 dist-test-slave-hllj krb5kdc[29289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388794, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:14.350700 29529 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:13:14.352366 29529 webserver.cc:460] Webserver started at http://127.28.152.130:43021/ using document root <none> and password file <none>
I20260926 02:13:14.353104 29529 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:14.353261 29529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:14.353547 29529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:14.356483 29529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "afe2b232efcf447e81aa8b3e8c48cc30"
format_stamp: "Formatted at 2026-09-26 02:13:14 on dist-test-slave-hllj"
server_key: "74d8bfc75a86c567dd56eda7f1ac14f1"
server_key_iv: "7c8f17c711c2675b686ceb1560c4e3a2"
server_key_version: "encryptionkey@0"
I20260926 02:13:14.357241 29529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "afe2b232efcf447e81aa8b3e8c48cc30"
format_stamp: "Formatted at 2026-09-26 02:13:14 on dist-test-slave-hllj"
server_key: "74d8bfc75a86c567dd56eda7f1ac14f1"
server_key_iv: "7c8f17c711c2675b686ceb1560c4e3a2"
server_key_version: "encryptionkey@0"
I20260926 02:13:14.362623 29529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:13:14.366403 29544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:14.367964 29529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:13:14.368124 29529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "afe2b232efcf447e81aa8b3e8c48cc30"
format_stamp: "Formatted at 2026-09-26 02:13:14 on dist-test-slave-hllj"
server_key: "74d8bfc75a86c567dd56eda7f1ac14f1"
server_key_iv: "7c8f17c711c2675b686ceb1560c4e3a2"
server_key_version: "encryptionkey@0"
I20260926 02:13:14.368317 29529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:14.388778 29529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:14.394171 29529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:14.394467 29529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:14.395534 29529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:14.396965 29529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:14.397043 29529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:14.397167 29529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:14.397233 29529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 02:13:14.431917 29529 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:41803
I20260926 02:13:14.432003 29656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:41803 every 8 connection(s)
I20260926 02:13:14.433290 29529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:13:14.442333 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 29529
I20260926 02:13:14.442538 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:13:14.443068 29282 external_mini_cluster.cc:1428] Setting key 5ef295ed70acef4df77cc78ddb863edb
Sep 26 02:13:14 dist-test-slave-hllj krb5kdc[29289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388794, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:14.455542 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:14.436008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:55191 (local address 127.28.152.190:40115)
0926 02:13:14.436218 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:14.436223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:14.438109 (+ 1886us) server_negotiation.cc:407] Connection header received
0926 02:13:14.440000 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:14.440006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:14.440098 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:14.440188 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:14.441900 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:14.442925 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:14.444866 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:14.445447 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:14.445586 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:14.449557 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:14.449562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:14.449566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:14.449602 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:14.452142 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:14.452970 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:14.452974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:14.452977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:14.453120 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:14.453639 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:14.453645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:14.453649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:14.454022 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:14.454184 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:14.454919 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 02:13:14.455168 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 02:13:14.456285 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:14.436584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:40115 (local address 127.28.152.130:55191)
0926 02:13:14.437916 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:14.437973 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:14.439683 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:14.440480 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:14.440489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:14.441042 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:14.441711 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:14.441731 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:14.443142 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:14.443148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:14.444714 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:14.444728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:14.445699 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:14.445704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:14.445932 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:14.446730 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:14.446761 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:14.449290 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:14.452296 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:14.452306 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:14.452327 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:14.452776 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:14.453277 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:14.453282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:14.453286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:14.453492 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:14.454148 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:14.454155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:14.454524 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 02:13:14.454766 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 02:13:14.455191 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":819,"threads_started":1}
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:13:14.457676 29657 heartbeater.cc:347] Connected to a master server at 127.28.152.190:40115
I20260926 02:13:14.457988 29657 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:14.459221 29657 heartbeater.cc:511] Master 127.28.152.190:40115 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
I20260926 02:13:14.460850 29334 ts_manager.cc:194] Registered new tserver with Master: afe2b232efcf447e81aa8b3e8c48cc30 (127.28.152.130:41803)
I20260926 02:13:14.461719 29334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:55191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:13:14.481679 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:40115
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:14.846639 29662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:14.846935 29662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:14.847059 29662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:14.854652 29662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:14.854781 29662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:14.854931 29662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:13:14.865578 29662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:40115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:14.867007 29662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:14.868327 29662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:14.882026 29667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:14.882045 29670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:14.882026 29668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:14.883831 29662 server_base.cc:1034] running on GCE node
I20260926 02:13:14.884598 29662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:14.885689 29662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:14.886907 29662 hybrid_clock.cc:648] HybridClock initialized: now 1790388794886856 us; error 52 us; skew 500 ppm
I20260926 02:13:14.887239 29662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:14 dist-test-slave-hllj krb5kdc[29289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388794, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:14.890473 29662 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:13:14.892108 29662 webserver.cc:460] Webserver started at http://127.28.152.131:35521/ using document root <none> and password file <none>
I20260926 02:13:14.892874 29662 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:14.893003 29662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:14.893366 29662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:14.896426 29662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "b8b85904a8ca4c768fa21f2562fcd61c"
format_stamp: "Formatted at 2026-09-26 02:13:14 on dist-test-slave-hllj"
server_key: "f4ae245dda908c0a1ea9e26c71ec16d5"
server_key_iv: "b31449c0e5c0390c962567517f98e25d"
server_key_version: "encryptionkey@0"
I20260926 02:13:14.897243 29662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "b8b85904a8ca4c768fa21f2562fcd61c"
format_stamp: "Formatted at 2026-09-26 02:13:14 on dist-test-slave-hllj"
server_key: "f4ae245dda908c0a1ea9e26c71ec16d5"
server_key_iv: "b31449c0e5c0390c962567517f98e25d"
server_key_version: "encryptionkey@0"
I20260926 02:13:14.902539 29662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:13:14.906342 29677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:14.907948 29662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 02:13:14.908133 29662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "b8b85904a8ca4c768fa21f2562fcd61c"
format_stamp: "Formatted at 2026-09-26 02:13:14 on dist-test-slave-hllj"
server_key: "f4ae245dda908c0a1ea9e26c71ec16d5"
server_key_iv: "b31449c0e5c0390c962567517f98e25d"
server_key_version: "encryptionkey@0"
I20260926 02:13:14.908350 29662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:14.925927 29662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:14.930171 29662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:14.930444 29662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:14.931423 29662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:14.932945 29662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:14.933028 29662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:14.933168 29662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:14.933228 29662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 02:13:14.950654 29523 heartbeater.cc:503] Master 127.28.152.190:40115 was elected leader, sending a full tablet report...
I20260926 02:13:14.982177 29662 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:43495
I20260926 02:13:14.982223 29789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:43495 every 8 connection(s)
I20260926 02:13:14.983428 29662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:13:14.989117 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 29662
I20260926 02:13:14.989318 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:13:14.989760 29282 external_mini_cluster.cc:1428] Setting key de840e77f0baa6203483c8465bc63cff
Sep 26 02:13:14 dist-test-slave-hllj krb5kdc[29289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388794, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:15.004211 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:14.986474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:34415 (local address 127.28.152.190:40115)
0926 02:13:14.987001 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:14.987007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:14.987928 (+ 921us) server_negotiation.cc:407] Connection header received
0926 02:13:14.989924 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:14.989930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:14.990015 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:14.990138 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:14.991770 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:14.992636 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:14.994303 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:14.994697 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:14.994796 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:14.998320 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:14.998325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:14.998328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:14.998362 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:15.001022 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.001920 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.001925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.001927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.002057 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.002523 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.002527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.002529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.002821 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:15.002955 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:15.003652 (+ 697us) server_negotiation.cc:299] Negotiation successful
0926 02:13:15.003901 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":274,"threads_started":1}
I20260926 02:13:15.005353 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:14.986978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:40115 (local address 127.28.152.131:34415)
0926 02:13:14.987755 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:14.987795 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:14.989618 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:14.990381 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:14.990391 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:14.990902 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:14.991588 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:14.991611 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:14.992810 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:14.992816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:14.994171 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:14.994184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:14.994918 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:14.994924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:14.995150 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:14.995832 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:14.995855 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:14.998100 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:15.001222 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:15.001231 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:15.001247 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:15.001768 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:15.002150 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:15.002154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:15.002156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:15.002357 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:15.002934 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:15.002940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:15.003292 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 02:13:15.003553 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 02:13:15.003963 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":315,"threads_started":1}
I20260926 02:13:15.006733 29790 heartbeater.cc:347] Connected to a master server at 127.28.152.190:40115
I20260926 02:13:15.007014 29790 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:15.008116 29790 heartbeater.cc:511] Master 127.28.152.190:40115 requested a full tablet report, sending...
I20260926 02:13:15.009546 29334 ts_manager.cc:194] Registered new tserver with Master: b8b85904a8ca4c768fa21f2562fcd61c (127.28.152.131:43495)
I20260926 02:13:15.010329 29334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:34415
I20260926 02:13:15.018520 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:13:15 dist-test-slave-hllj krb5kdc[29289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388795, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:13:15 dist-test-slave-hllj krb5kdc[29289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388795, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:13:15.064095 29798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:15.047942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.28.152.129:44547)
0926 02:13:15.048677 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:15.048684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:15.048721 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:13:15.048784 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:15.048791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:15.049013 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:15.049202 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:15.050662 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:15.051824 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:15.053271 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:15.053907 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:15.054031 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:15.058603 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:15.058624 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:15.058639 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:15.058684 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:15.061340 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.062022 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.062030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.062032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.062160 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.062557 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.062561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.062563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.062915 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:15.063189 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:15.063500 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 02:13:15.063766 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":452,"threads_started":1}
W20260926 02:13:15.065378 29438 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56404
Sep 26 02:13:15 dist-test-slave-hllj krb5kdc[29289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:13:15 dist-test-slave-hllj krb5kdc[29289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:13:15.107856 29798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:15.093681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.28.152.129:44547)
0926 02:13:15.093906 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:15.093912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:15.094108 (+ 196us) server_negotiation.cc:407] Connection header received
0926 02:13:15.094329 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:15.094333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:15.094397 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:15.094487 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:15.095900 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:15.096593 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:15.098456 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:15.098935 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:15.099058 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:15.103003 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:15.103008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:15.103012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:15.103051 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:15.105415 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.106053 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.106057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.106060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.106182 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.106542 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.106546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.106550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.106812 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:15.107024 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:15.107329 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 02:13:15.107541 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:13:15.109489 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 29395
I20260926 02:13:15.123178 29798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:15.110341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.28.152.129:44547)
0926 02:13:15.110535 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:15.110541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:15.110818 (+ 277us) server_negotiation.cc:407] Connection header received
0926 02:13:15.110942 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:15.110946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:15.111002 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:15.111076 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:15.112280 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:15.113073 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:15.114389 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:15.114845 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:15.114957 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:15.117299 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:15.117304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:15.117308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:15.117344 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:15.120302 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.121068 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.121073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.121076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.121287 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.121764 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.121768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.121770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.122062 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:15.122260 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:15.122649 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 02:13:15.122832 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:13:15.123998 29518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:15.205888 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 29395
I20260926 02:13:15.219163 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 29529
Sep 26 02:13:15 dist-test-slave-hllj krb5kdc[29289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:13:15.235158 29805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:15.220025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.28.152.130:41803)
0926 02:13:15.220488 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:15.220496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:15.220521 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:13:15.220581 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:15.220587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:15.220801 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:15.220956 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:15.222169 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:15.223305 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:15.224616 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:15.225236 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:15.225354 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:15.229411 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:15.229424 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:15.229442 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:15.229479 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:15.232132 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.232812 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.232820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.232823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.232980 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.233533 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.233537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.233539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.233892 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:15.234181 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:15.234500 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 02:13:15.234758 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":220,"threads_started":1}
I20260926 02:13:15.236673 29652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:15.322201 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 29529
I20260926 02:13:15.340756 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 29662
Sep 26 02:13:15 dist-test-slave-hllj krb5kdc[29289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:13:15.356462 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:15.341633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.28.152.131:43495)
0926 02:13:15.342240 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:15.342247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:15.342275 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:13:15.342335 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:15.342342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:15.342562 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:15.342699 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:15.343933 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:15.344989 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:15.346259 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:15.346806 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:15.346910 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:15.350758 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:15.350771 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:15.350787 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:15.350823 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:15.353860 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.354613 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.354621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.354624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.354776 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:15.355104 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:15.355108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:15.355110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:15.355489 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:15.355732 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:15.355975 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 02:13:15.356186 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":295,"threads_started":1}
I20260926 02:13:15.357797 29785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:15.438591 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 29662
I20260926 02:13:15.452050 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 29302
I20260926 02:13:15.453193 29364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:15.553159 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 29302
2026-09-26T02:13:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3878 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:13:15 dist-test-slave-hllj krb5kdc[29812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:13:15 dist-test-slave-hllj krb5kdc[29812](info): set up 2 sockets
Sep 26 02:13:15 dist-test-slave-hllj krb5kdc[29812](info): commencing operation
krb5kdc: starting...
W20260926 02:13:16.550485 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:13:16 dist-test-slave-hllj krb5kdc[29812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:13:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:13:16.728482 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:33003
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:33003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:17.049151 29825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:17.049458 29825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:17.049566 29825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:17.057230 29825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:17.057422 29825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:17.057482 29825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:17.057564 29825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:17.057627 29825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:13:17.068089 29825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:33003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:33003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:17.069530 29825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:17.070940 29825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:17.082899 29831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:17.082965 29833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:17.082932 29830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:17.083807 29825 server_base.cc:1034] running on GCE node
I20260926 02:13:17.084616 29825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:17.086189 29825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:17.087409 29825 hybrid_clock.cc:648] HybridClock initialized: now 1790388797087357 us; error 45 us; skew 500 ppm
I20260926 02:13:17.087744 29825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:17 dist-test-slave-hllj krb5kdc[29812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388797, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:17.090816 29825 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:13:17.092330 29825 webserver.cc:460] Webserver started at http://127.28.152.190:45525/ using document root <none> and password file <none>
I20260926 02:13:17.092990 29825 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:17.093088 29825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:17.093425 29825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:17.096392 29825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "f363fa58957b4e4c91f53ef2aad24508"
format_stamp: "Formatted at 2026-09-26 02:13:17 on dist-test-slave-hllj"
server_key: "d712bc844b962616133506b766b0e844"
server_key_iv: "f981eb12bc0289b6845faaed1761f2a5"
server_key_version: "encryptionkey@0"
I20260926 02:13:17.097105 29825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "f363fa58957b4e4c91f53ef2aad24508"
format_stamp: "Formatted at 2026-09-26 02:13:17 on dist-test-slave-hllj"
server_key: "d712bc844b962616133506b766b0e844"
server_key_iv: "f981eb12bc0289b6845faaed1761f2a5"
server_key_version: "encryptionkey@0"
I20260926 02:13:17.102430 29825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:13:17.106019 29840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:17.107435 29825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:17.107585 29825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "f363fa58957b4e4c91f53ef2aad24508"
format_stamp: "Formatted at 2026-09-26 02:13:17 on dist-test-slave-hllj"
server_key: "d712bc844b962616133506b766b0e844"
server_key_iv: "f981eb12bc0289b6845faaed1761f2a5"
server_key_version: "encryptionkey@0"
I20260926 02:13:17.107755 29825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:17.135294 29825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:17.139842 29825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:17.140071 29825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:17.157094 29825 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:33003
I20260926 02:13:17.157161 29891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:33003 every 8 connection(s)
I20260926 02:13:17.158550 29825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:17.162498 29892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:17.165442 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 29825
I20260926 02:13:17.165671 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:17.166442 29282 external_mini_cluster.cc:1428] Setting key fd3896ae61bc0c3c391f2c9d4c9ac26e
I20260926 02:13:17.175403 29892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508: Bootstrap starting.
I20260926 02:13:17.179239 29892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:17.180574 29892 log.cc:824] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:13:17 dist-test-slave-hllj krb5kdc[29812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:17.183743 29892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508: No bootstrap required, opened a new log
I20260926 02:13:17.190141 29895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:17.168677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.28.152.190:33003)
0926 02:13:17.169512 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:17.169526 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:17.169573 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:13:17.171237 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:17.171261 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:17.171722 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:17.172090 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:17.173923 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:17.175270 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:17.177007 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:17.177764 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:17.177891 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:17.182731 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:17.182739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:17.182756 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:17.182793 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:17.185772 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:17.186600 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:17.186607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:17.186613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:17.186785 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:17.187195 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:17.187200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:17.187202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:17.187817 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:17.188036 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:17.188616 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 02:13:17.188989 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":407,"threads_started":1}
I20260926 02:13:17.192708 29892 raft_consensus.cc:348] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f363fa58957b4e4c91f53ef2aad24508" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33003 } }
I20260926 02:13:17.193145 29892 raft_consensus.cc:374] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:17.193264 29892 raft_consensus.cc:729] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f363fa58957b4e4c91f53ef2aad24508, State: Initialized, Role: FOLLOWER
I20260926 02:13:17.193830 29892 consensus_queue.cc:260] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f363fa58957b4e4c91f53ef2aad24508" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33003 } }
I20260926 02:13:17.194078 29892 raft_consensus.cc:388] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:17.194199 29892 raft_consensus.cc:482] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:17.194350 29892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:17.196455 29892 raft_consensus.cc:504] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f363fa58957b4e4c91f53ef2aad24508" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33003 } }
I20260926 02:13:17.196939 29892 leader_election.cc:302] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f363fa58957b4e4c91f53ef2aad24508; no voters:
I20260926 02:13:17.197634 29892 leader_election.cc:288] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:17.197772 29897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:17.198453 29897 raft_consensus.cc:686] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [term 1 LEADER]: Becoming Leader. State: Replica: f363fa58957b4e4c91f53ef2aad24508, State: Running, Role: LEADER
I20260926 02:13:17.198994 29897 consensus_queue.cc:237] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f363fa58957b4e4c91f53ef2aad24508" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33003 } }
I20260926 02:13:17.199764 29892 sys_catalog.cc:564] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:17.202023 29898 sys_catalog.cc:455] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f363fa58957b4e4c91f53ef2aad24508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f363fa58957b4e4c91f53ef2aad24508" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33003 } } }
I20260926 02:13:17.202426 29898 sys_catalog.cc:458] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:17.203083 29899 sys_catalog.cc:455] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f363fa58957b4e4c91f53ef2aad24508. Latest consensus state: current_term: 1 leader_uuid: "f363fa58957b4e4c91f53ef2aad24508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f363fa58957b4e4c91f53ef2aad24508" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33003 } } }
I20260926 02:13:17.203413 29899 sys_catalog.cc:458] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:17.204036 29904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:17.209352 29904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:17.219113 29904 catalog_manager.cc:1347] Generated new cluster ID: b5a58d346c6242d4b9988f2d4fbb169b
I20260926 02:13:17.219246 29904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:17.230202 29904 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:17.232110 29904 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:17.247875 29904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f363fa58957b4e4c91f53ef2aad24508: Generated new TSK 0
I20260926 02:13:17.248826 29904 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:13:17.305114 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33003
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:17.651979 29918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:17.652343 29918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:17.652503 29918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:17.659963 29918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:17.660100 29918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:17.660254 29918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:13:17.670544 29918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:17.671911 29918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:17.673424 29918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:17.686941 29923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:17.686966 29926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:17.686894 29924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:17.687911 29918 server_base.cc:1034] running on GCE node
I20260926 02:13:17.688597 29918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:17.689656 29918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:17.690858 29918 hybrid_clock.cc:648] HybridClock initialized: now 1790388797690812 us; error 50 us; skew 500 ppm
I20260926 02:13:17.691166 29918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:17 dist-test-slave-hllj krb5kdc[29812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388797, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:17.694330 29918 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:13:17.695757 29918 webserver.cc:460] Webserver started at http://127.28.152.129:38591/ using document root <none> and password file <none>
I20260926 02:13:17.696450 29918 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:17.696595 29918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:17.696880 29918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:17.699991 29918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "aab09611a057443b8fe9647d89e2216d"
format_stamp: "Formatted at 2026-09-26 02:13:17 on dist-test-slave-hllj"
server_key: "8d6b549cf56c58f17317ef083dbe83da"
server_key_iv: "4b549a8815e3a604726d16ecbc6b777e"
server_key_version: "encryptionkey@0"
I20260926 02:13:17.700842 29918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "aab09611a057443b8fe9647d89e2216d"
format_stamp: "Formatted at 2026-09-26 02:13:17 on dist-test-slave-hllj"
server_key: "8d6b549cf56c58f17317ef083dbe83da"
server_key_iv: "4b549a8815e3a604726d16ecbc6b777e"
server_key_version: "encryptionkey@0"
I20260926 02:13:17.706139 29918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 02:13:17.709774 29933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:17.711223 29918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 02:13:17.711382 29918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "aab09611a057443b8fe9647d89e2216d"
format_stamp: "Formatted at 2026-09-26 02:13:17 on dist-test-slave-hllj"
server_key: "8d6b549cf56c58f17317ef083dbe83da"
server_key_iv: "4b549a8815e3a604726d16ecbc6b777e"
server_key_version: "encryptionkey@0"
I20260926 02:13:17.711577 29918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:17.723932 29918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:17.728262 29918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:17.728549 29918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:17.729544 29918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:17.730906 29918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:17.730983 29918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:17.731098 29918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:17.731153 29918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:17.761441 29918 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:42499
I20260926 02:13:17.761499 30045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:42499 every 8 connection(s)
I20260926 02:13:17.762758 29918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:13:17.764369 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 29918
I20260926 02:13:17.764566 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:13:17.765198 29282 external_mini_cluster.cc:1428] Setting key a7417eb6df4672db593dc5221794a9f0
Sep 26 02:13:17 dist-test-slave-hllj krb5kdc[29812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388797, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
I20260926 02:13:17.786350 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:17.765403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:37727 (local address 127.28.152.190:33003)
0926 02:13:17.765983 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:17.765990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:17.767491 (+ 1501us) server_negotiation.cc:407] Connection header received
0926 02:13:17.769641 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:17.769646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:17.769743 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:17.769840 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:17.771616 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:17.772371 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:17.774719 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:17.775145 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:17.775246 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:17.779607 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:17.779614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:17.779617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:17.779654 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:17.782671 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:17.783609 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:17.783614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:17.783617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:17.783849 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:17.784399 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:17.784405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:17.784408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:17.784722 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:17.784896 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:17.785735 (+ 839us) server_negotiation.cc:299] Negotiation successful
0926 02:13:17.785990 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":216,"threads_started":1}
I20260926 02:13:17.787199 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:17.766135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33003 (local address 127.28.152.129:37727)
0926 02:13:17.767334 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:17.767383 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:17.769354 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:17.770101 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:17.770110 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:17.770653 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:17.771411 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:17.771438 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:17.772534 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:17.772540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:17.774557 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:17.774572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:17.775366 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:17.775372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:17.775669 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:17.776443 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:17.776474 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:17.779255 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:17.782853 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:17.782866 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:17.782933 (+ 67us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:17.783440 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:17.783968 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:17.783973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:17.783977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:17.784194 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:17.784874 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:17.784881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:17.785320 (+ 439us) client_negotiation.cc:769] Sending connection context
0926 02:13:17.785571 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 02:13:17.785998 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"thread_start_us":666,"threads_started":1}
I20260926 02:13:17.788756 30046 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33003
I20260926 02:13:17.789059 30046 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:17.789837 30046 heartbeater.cc:511] Master 127.28.152.190:33003 requested a full tablet report, sending...
I20260926 02:13:17.791882 29857 ts_manager.cc:194] Registered new tserver with Master: aab09611a057443b8fe9647d89e2216d (127.28.152.129:42499)
I20260926 02:13:17.793466 29857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:37727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:13:17.808737 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33003
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:18.134125 30052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:18.134411 30052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:18.134557 30052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:18.141665 30052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:18.141783 30052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:18.141903 30052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:13:18.151712 30052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:18.153069 30052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:18.154533 30052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:18.169185 30058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:18.169565 30060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:18.169162 30057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:18.170197 30052 server_base.cc:1034] running on GCE node
I20260926 02:13:18.170857 30052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:18.171814 30052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:18.173022 30052 hybrid_clock.cc:648] HybridClock initialized: now 1790388798172962 us; error 52 us; skew 500 ppm
I20260926 02:13:18.173404 30052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:18 dist-test-slave-hllj krb5kdc[29812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388798, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:18.176460 30052 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:13:18.177942 30052 webserver.cc:460] Webserver started at http://127.28.152.130:41269/ using document root <none> and password file <none>
I20260926 02:13:18.178620 30052 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:18.178735 30052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:18.179036 30052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:18.182052 30052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "bbe4da2bd3044d4294446c8101382d70"
format_stamp: "Formatted at 2026-09-26 02:13:18 on dist-test-slave-hllj"
server_key: "3b9c74fe6911aa97859192da3c713047"
server_key_iv: "cd7fe289af3f721c79aa9085d644e598"
server_key_version: "encryptionkey@0"
I20260926 02:13:18.182821 30052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "bbe4da2bd3044d4294446c8101382d70"
format_stamp: "Formatted at 2026-09-26 02:13:18 on dist-test-slave-hllj"
server_key: "3b9c74fe6911aa97859192da3c713047"
server_key_iv: "cd7fe289af3f721c79aa9085d644e598"
server_key_version: "encryptionkey@0"
I20260926 02:13:18.188213 30052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:13:18.191974 30067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:18.193529 30052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:13:18.193696 30052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "bbe4da2bd3044d4294446c8101382d70"
format_stamp: "Formatted at 2026-09-26 02:13:18 on dist-test-slave-hllj"
server_key: "3b9c74fe6911aa97859192da3c713047"
server_key_iv: "cd7fe289af3f721c79aa9085d644e598"
server_key_version: "encryptionkey@0"
I20260926 02:13:18.193898 30052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:18.213033 30052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:18.217453 30052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:18.217730 30052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:18.218696 30052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:18.220055 30052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:18.220131 30052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:18.220248 30052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:18.220300 30052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:18.250315 30052 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:45485
I20260926 02:13:18.250360 30179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:45485 every 8 connection(s)
I20260926 02:13:18.251605 30052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:13:18.256351 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 30052
I20260926 02:13:18.256528 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:13:18.256947 29282 external_mini_cluster.cc:1428] Setting key 11b65ed4433b80bdafbbb8f0165b1a6d
Sep 26 02:13:18 dist-test-slave-hllj krb5kdc[29812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388798, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:13:18.271014 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:18.254049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:43583 (local address 127.28.152.190:33003)
0926 02:13:18.254201 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:18.254206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:18.255420 (+ 1214us) server_negotiation.cc:407] Connection header received
0926 02:13:18.257240 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:18.257245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:18.257330 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:18.257423 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:18.259222 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.259922 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.261492 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.261907 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.262012 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:18.265410 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:18.265415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:18.265418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:18.265453 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:18.267977 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:18.268712 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:18.268717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:18.268719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:18.268840 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:18.269299 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:18.269303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:18.269305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:18.269592 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:18.269723 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:18.270449 (+ 726us) server_negotiation.cc:299] Negotiation successful
0926 02:13:18.270679 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:13:18.272102 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:18.254531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33003 (local address 127.28.152.130:43583)
0926 02:13:18.255213 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:18.255250 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:18.256947 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:18.257668 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:18.257677 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:18.258356 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:18.259061 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:18.259080 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.260069 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.260074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:18.261360 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:18.261374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.262118 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.262125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:18.262337 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:18.262996 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:18.263025 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:18.265120 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:18.268125 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:18.268135 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:18.268155 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:18.268585 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:18.268946 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:18.268951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:18.268955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:18.269190 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:18.269714 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:18.269722 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:18.270088 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 02:13:18.270335 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 02:13:18.270913 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":260,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
I20260926 02:13:18.273676 30180 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33003
I20260926 02:13:18.274008 30180 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:18.274748 30180 heartbeater.cc:511] Master 127.28.152.190:33003 requested a full tablet report, sending...
I20260926 02:13:18.276315 29857 ts_manager.cc:194] Registered new tserver with Master: bbe4da2bd3044d4294446c8101382d70 (127.28.152.130:45485)
I20260926 02:13:18.277377 29857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:43583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:13:18.297207 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33003
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:18.626076 30185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:18.626400 30185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:18.626542 30185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:18.633819 30185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:18.633955 30185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:18.634116 30185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:13:18.643850 30185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:18.645200 30185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:18.646638 30185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:18.660164 30190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:18.660131 30191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:18.660099 30193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:18.660985 30185 server_base.cc:1034] running on GCE node
I20260926 02:13:18.661659 30185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:18.662688 30185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:18.663923 30185 hybrid_clock.cc:648] HybridClock initialized: now 1790388798663847 us; error 64 us; skew 500 ppm
I20260926 02:13:18.664286 30185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:18 dist-test-slave-hllj krb5kdc[29812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388798, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:18.667645 30185 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:13:18.669243 30185 webserver.cc:460] Webserver started at http://127.28.152.131:39669/ using document root <none> and password file <none>
I20260926 02:13:18.669957 30185 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:18.670070 30185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:18.670398 30185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:18.673559 30185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "e446651a93924539908dd74094343d20"
format_stamp: "Formatted at 2026-09-26 02:13:18 on dist-test-slave-hllj"
server_key: "a7ccb9ae18a4972db6909e9dd70256d8"
server_key_iv: "34c791cbdf43cb2c758e90502c8994fd"
server_key_version: "encryptionkey@0"
I20260926 02:13:18.674360 30185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "e446651a93924539908dd74094343d20"
format_stamp: "Formatted at 2026-09-26 02:13:18 on dist-test-slave-hllj"
server_key: "a7ccb9ae18a4972db6909e9dd70256d8"
server_key_iv: "34c791cbdf43cb2c758e90502c8994fd"
server_key_version: "encryptionkey@0"
I20260926 02:13:18.679936 30185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 02:13:18.683792 30200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:18.685369 30185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:13:18.685541 30185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "e446651a93924539908dd74094343d20"
format_stamp: "Formatted at 2026-09-26 02:13:18 on dist-test-slave-hllj"
server_key: "a7ccb9ae18a4972db6909e9dd70256d8"
server_key_iv: "34c791cbdf43cb2c758e90502c8994fd"
server_key_version: "encryptionkey@0"
I20260926 02:13:18.685734 30185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:18.709110 30185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:18.713685 30185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:18.713953 30185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:18.714968 30185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:18.716434 30185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:18.716521 30185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:18.716614 30185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:18.716655 30185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:18.745208 30185 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:35151
I20260926 02:13:18.745306 30312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:35151 every 8 connection(s)
I20260926 02:13:18.746515 30185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:13:18.748741 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 30185
I20260926 02:13:18.748936 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:13:18.749531 29282 external_mini_cluster.cc:1428] Setting key 8de69384328ebd079cbab4b7fd287cf2
Sep 26 02:13:18 dist-test-slave-hllj krb5kdc[29812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388798, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:18.766911 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:18.749288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:43567 (local address 127.28.152.190:33003)
0926 02:13:18.749461 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:18.749465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:18.750809 (+ 1344us) server_negotiation.cc:407] Connection header received
0926 02:13:18.752798 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:18.752802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:18.752864 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:18.752933 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:18.754526 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.755280 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.756807 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.757274 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.757375 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:18.760748 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:18.760754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:18.760757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:18.760791 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:18.763661 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:18.764538 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:18.764544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:18.764549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:18.764699 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:18.765198 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:18.765203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:18.765205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:18.765531 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:18.765644 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:18.766503 (+ 859us) server_negotiation.cc:299] Negotiation successful
0926 02:13:18.766673 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:13:18.767975 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:18.749910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33003 (local address 127.28.152.131:43567)
0926 02:13:18.750664 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:18.750702 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:18.752541 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:18.753222 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:18.753231 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:18.753774 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:18.754357 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:18.754375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.755390 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.755394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:18.756670 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:18.756682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.757478 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.757484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:18.757684 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:18.758316 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:18.758339 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:18.760524 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:18.763800 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:18.763810 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:18.763828 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:18.764396 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:18.764802 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:18.764807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:18.764811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:18.765056 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:18.765648 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:18.765657 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:18.766092 (+ 435us) client_negotiation.cc:769] Sending connection context
0926 02:13:18.766384 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 02:13:18.766927 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":288,"threads_started":1}
I20260926 02:13:18.769296 30313 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33003
I20260926 02:13:18.769580 30313 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:18.770258 30313 heartbeater.cc:511] Master 127.28.152.190:33003 requested a full tablet report, sending...
I20260926 02:13:18.771654 29857 ts_manager.cc:194] Registered new tserver with Master: e446651a93924539908dd74094343d20 (127.28.152.131:35151)
I20260926 02:13:18.772444 29857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:43567
I20260926 02:13:18.777457 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:13:18.796733 30046 heartbeater.cc:503] Master 127.28.152.190:33003 was elected leader, sending a full tablet report...
I20260926 02:13:18.796934 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:18.784074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.28.152.190:33003)
0926 02:13:18.784233 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:18.784238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:18.784731 (+ 493us) server_negotiation.cc:407] Connection header received
0926 02:13:18.784857 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:18.784861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:18.784929 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:18.785014 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:18.786486 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.787193 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.788481 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.788895 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.788999 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:18.791640 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:18.791645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:18.791648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:18.791685 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:18.794423 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:18.795036 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:18.795041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:18.795044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:18.795175 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:18.795534 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:18.795538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:18.795541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:18.795872 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:18.796076 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:18.796471 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 02:13:18.796650 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 02:13:18.804033 29856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:13:18.811331 29856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:13:18.845467 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:18.825477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.28.152.130:45485)
0926 02:13:18.837062 (+ 11585us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:18.837071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:18.837621 (+ 550us) server_negotiation.cc:407] Connection header received
0926 02:13:18.837949 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:18.837957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:18.838157 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:18.838327 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:18.839101 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.840475 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.842778 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.844273 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.844626 (+ 353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:18.844829 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:18.844903 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 02:13:18.845083 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11424,"thread_start_us":10768,"threads_started":1}
I20260926 02:13:18.845484 30325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:18.824920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:45485 (local address 127.0.0.1:43476)
0926 02:13:18.837495 (+ 12575us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:18.837513 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:18.837728 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:18.838567 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:18.838571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:18.838592 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:18.838917 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:18.838929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.840596 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.840601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:18.842646 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:18.842656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.844496 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.844502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:18.844687 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:18.844739 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:13:18.844859 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 02:13:18.845072 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12442,"thread_start_us":10135,"threads_started":1}
I20260926 02:13:18.845906 30328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:18.824186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42499 (local address 127.0.0.1:41674)
0926 02:13:18.834544 (+ 10358us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:18.834614 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:18.834838 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:18.837627 (+ 2789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:18.837635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:18.837667 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:18.838162 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:18.838178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.840094 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.840099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:18.842500 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:18.842516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.845017 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.845023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:18.845212 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:18.845266 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 02:13:18.845351 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 02:13:18.845533 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10001,"mutex_wait_us":6,"thread_start_us":13043,"threads_started":1}
I20260926 02:13:18.846587 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:18.824375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.28.152.129:42499)
0926 02:13:18.836887 (+ 12512us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:18.836898 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:18.836932 (+ 34us) server_negotiation.cc:407] Connection header received
0926 02:13:18.837046 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:18.837054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:18.837362 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:18.838383 (+ 1021us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:18.838588 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.839966 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.842658 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.844580 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.844748 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:18.844877 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:18.846121 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0926 02:13:18.846286 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12289,"thread_start_us":12141,"threads_started":1}
I20260926 02:13:18.847747 30327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:18.824411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:35151 (local address 127.0.0.1:39094)
0926 02:13:18.836158 (+ 11747us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:18.836178 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:18.836299 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:18.837859 (+ 1560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:18.837864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:18.837887 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:18.838371 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:18.838397 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.840000 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.840004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:18.844088 (+ 4084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:18.844100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.847027 (+ 2927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.847034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:18.847184 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:18.847206 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 02:13:18.847284 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 02:13:18.847407 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11609,"mutex_wait_us":2163,"thread_start_us":11484,"threads_started":1}
I20260926 02:13:18.847988 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:18.825017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.28.152.131:35151)
0926 02:13:18.837258 (+ 12241us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:18.837266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:18.837300 (+ 34us) server_negotiation.cc:407] Connection header received
0926 02:13:18.837410 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:18.837417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:18.837650 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:18.838554 (+ 904us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:18.838817 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.839865 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.844291 (+ 4426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:18.846276 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:18.846507 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:18.846857 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:18.847344 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 02:13:18.847518 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12064,"thread_start_us":11922,"threads_started":1}
I20260926 02:13:18.850114 29981 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9b0de9127e46d59a07db2566a020ce (DEFAULT_TABLE table=test-table [id=fc21972cf0e8472095c75621eb9d81e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:18.849989 30115 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9b0de9127e46d59a07db2566a020ce (DEFAULT_TABLE table=test-table [id=fc21972cf0e8472095c75621eb9d81e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:18.851428 29981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9b0de9127e46d59a07db2566a020ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:18.851459 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9b0de9127e46d59a07db2566a020ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:18.851213 30248 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9b0de9127e46d59a07db2566a020ce (DEFAULT_TABLE table=test-table [id=fc21972cf0e8472095c75621eb9d81e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:18.852327 30248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9b0de9127e46d59a07db2566a020ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:18.864363 30331 tablet_bootstrap.cc:492] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20: Bootstrap starting.
I20260926 02:13:18.866580 30332 tablet_bootstrap.cc:492] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70: Bootstrap starting.
I20260926 02:13:18.867820 30331 tablet_bootstrap.cc:654] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:18.869961 30331 log.cc:824] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:18.870193 30332 tablet_bootstrap.cc:654] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:18.871560 30332 log.cc:824] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:18.871950 30333 tablet_bootstrap.cc:492] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d: Bootstrap starting.
I20260926 02:13:18.874539 30332 tablet_bootstrap.cc:492] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70: No bootstrap required, opened a new log
I20260926 02:13:18.874634 30331 tablet_bootstrap.cc:492] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20: No bootstrap required, opened a new log
I20260926 02:13:18.874826 30332 ts_tablet_manager.cc:1397] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:13:18.874928 30331 ts_tablet_manager.cc:1397] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 02:13:18.875823 30333 tablet_bootstrap.cc:654] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:18.877110 30333 log.cc:824] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:18.881508 30333 tablet_bootstrap.cc:492] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d: No bootstrap required, opened a new log
I20260926 02:13:18.881806 30333 ts_tablet_manager.cc:1397] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 02:13:18.883209 30332 raft_consensus.cc:348] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab09611a057443b8fe9647d89e2216d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42499 } } peers { permanent_uuid: "e446651a93924539908dd74094343d20" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35151 } } peers { permanent_uuid: "bbe4da2bd3044d4294446c8101382d70" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45485 } }
I20260926 02:13:18.883594 30332 raft_consensus.cc:374] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:18.883692 30332 raft_consensus.cc:729] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe4da2bd3044d4294446c8101382d70, State: Initialized, Role: FOLLOWER
I20260926 02:13:18.884287 30332 consensus_queue.cc:260] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab09611a057443b8fe9647d89e2216d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42499 } } peers { permanent_uuid: "e446651a93924539908dd74094343d20" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35151 } } peers { permanent_uuid: "bbe4da2bd3044d4294446c8101382d70" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45485 } }
I20260926 02:13:18.884768 30331 raft_consensus.cc:348] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab09611a057443b8fe9647d89e2216d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42499 } } peers { permanent_uuid: "e446651a93924539908dd74094343d20" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35151 } } peers { permanent_uuid: "bbe4da2bd3044d4294446c8101382d70" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45485 } }
I20260926 02:13:18.885391 30331 raft_consensus.cc:374] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:18.885471 30180 heartbeater.cc:503] Master 127.28.152.190:33003 was elected leader, sending a full tablet report...
I20260926 02:13:18.885501 30331 raft_consensus.cc:729] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e446651a93924539908dd74094343d20, State: Initialized, Role: FOLLOWER
I20260926 02:13:18.886082 30332 ts_tablet_manager.cc:1428] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 02:13:18.886216 30331 consensus_queue.cc:260] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab09611a057443b8fe9647d89e2216d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42499 } } peers { permanent_uuid: "e446651a93924539908dd74094343d20" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35151 } } peers { permanent_uuid: "bbe4da2bd3044d4294446c8101382d70" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45485 } }
I20260926 02:13:18.887283 30313 heartbeater.cc:503] Master 127.28.152.190:33003 was elected leader, sending a full tablet report...
I20260926 02:13:18.888293 30331 ts_tablet_manager.cc:1428] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 02:13:18.892349 30333 raft_consensus.cc:348] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab09611a057443b8fe9647d89e2216d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42499 } } peers { permanent_uuid: "e446651a93924539908dd74094343d20" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35151 } } peers { permanent_uuid: "bbe4da2bd3044d4294446c8101382d70" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45485 } }
I20260926 02:13:18.892779 30333 raft_consensus.cc:374] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:18.892901 30333 raft_consensus.cc:729] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab09611a057443b8fe9647d89e2216d, State: Initialized, Role: FOLLOWER
I20260926 02:13:18.893508 30333 consensus_queue.cc:260] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab09611a057443b8fe9647d89e2216d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42499 } } peers { permanent_uuid: "e446651a93924539908dd74094343d20" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35151 } } peers { permanent_uuid: "bbe4da2bd3044d4294446c8101382d70" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45485 } }
I20260926 02:13:18.895160 30333 ts_tablet_manager.cc:1428] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d: Time spent starting tablet: real 0.013s user 0.003s sys 0.009s
W20260926 02:13:18.998092 30314 tablet.cc:2369] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:13:19.003516 30181 tablet.cc:2369] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:13:19.015133 30047 tablet.cc:2369] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:13:19.185228 30338 raft_consensus.cc:482] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:19.185597 30338 raft_consensus.cc:504] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab09611a057443b8fe9647d89e2216d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42499 } } peers { permanent_uuid: "e446651a93924539908dd74094343d20" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35151 } } peers { permanent_uuid: "bbe4da2bd3044d4294446c8101382d70" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45485 } }
I20260926 02:13:19.187173 30338 leader_election.cc:288] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aab09611a057443b8fe9647d89e2216d (127.28.152.129:42499), bbe4da2bd3044d4294446c8101382d70 (127.28.152.130:45485)
I20260926 02:13:19.194732 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.187694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:59911 (local address 127.28.152.130:45485)
0926 02:13:19.187876 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:19.187882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:19.188107 (+ 225us) server_negotiation.cc:407] Connection header received
0926 02:13:19.188213 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:19.188218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:19.188347 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:19.188440 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:19.188977 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.190033 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.192423 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.193648 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.193824 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:19.193902 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:19.194385 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 02:13:19.194512 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:13:19.194729 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.187678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:41875 (local address 127.28.152.129:42499)
0926 02:13:19.187876 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:19.187882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:19.187904 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:13:19.188094 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:19.188100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:19.188180 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:19.188300 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:19.188685 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.189811 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.192371 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.193693 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.193836 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:19.193902 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:19.194339 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 02:13:19.194474 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:13:19.194826 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.187434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42499 (local address 127.28.152.131:41875)
0926 02:13:19.187750 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:19.187773 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:19.187903 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:19.188307 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:19.188311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:19.188332 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:19.188513 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:19.188525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.189948 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.189953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:19.192168 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:19.192185 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.193850 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.193857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:19.194104 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:19.194203 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 02:13:19.194300 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 02:13:19.194471 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":34}
I20260926 02:13:19.194876 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.187476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:45485 (local address 127.28.152.131:59911)
0926 02:13:19.187995 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:19.188010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:19.188116 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:19.188467 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:19.188472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:19.188490 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:19.188831 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:19.188840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.190157 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.190161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:19.192188 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:19.192197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.193854 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.193859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:19.194111 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:19.194203 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 02:13:19.194300 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 02:13:19.194470 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":173,"thread_start_us":522,"threads_started":1}
I20260926 02:13:19.196031 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9b0de9127e46d59a07db2566a020ce" candidate_uuid: "e446651a93924539908dd74094343d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe4da2bd3044d4294446c8101382d70" is_pre_election: true
I20260926 02:13:19.196031 30001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9b0de9127e46d59a07db2566a020ce" candidate_uuid: "e446651a93924539908dd74094343d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab09611a057443b8fe9647d89e2216d" is_pre_election: true
I20260926 02:13:19.196609 30135 raft_consensus.cc:2454] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e446651a93924539908dd74094343d20 in term 0.
I20260926 02:13:19.196609 30001 raft_consensus.cc:2454] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e446651a93924539908dd74094343d20 in term 0.
I20260926 02:13:19.197369 30201 leader_election.cc:302] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: aab09611a057443b8fe9647d89e2216d, bbe4da2bd3044d4294446c8101382d70, e446651a93924539908dd74094343d20; no voters:
I20260926 02:13:19.197763 30338 raft_consensus.cc:2781] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:19.197890 30338 raft_consensus.cc:482] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:19.197985 30338 raft_consensus.cc:3037] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:19.200146 30338 raft_consensus.cc:504] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab09611a057443b8fe9647d89e2216d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42499 } } peers { permanent_uuid: "e446651a93924539908dd74094343d20" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35151 } } peers { permanent_uuid: "bbe4da2bd3044d4294446c8101382d70" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45485 } }
I20260926 02:13:19.200763 30338 leader_election.cc:288] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [CANDIDATE]: Term 1 election: Requested vote from peers aab09611a057443b8fe9647d89e2216d (127.28.152.129:42499), bbe4da2bd3044d4294446c8101382d70 (127.28.152.130:45485)
I20260926 02:13:19.201267 30001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9b0de9127e46d59a07db2566a020ce" candidate_uuid: "e446651a93924539908dd74094343d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab09611a057443b8fe9647d89e2216d"
I20260926 02:13:19.201323 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9b0de9127e46d59a07db2566a020ce" candidate_uuid: "e446651a93924539908dd74094343d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe4da2bd3044d4294446c8101382d70"
I20260926 02:13:19.201529 30001 raft_consensus.cc:3037] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:19.201541 30135 raft_consensus.cc:3037] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:19.203658 30135 raft_consensus.cc:2454] T 0a9b0de9127e46d59a07db2566a020ce P bbe4da2bd3044d4294446c8101382d70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e446651a93924539908dd74094343d20 in term 1.
I20260926 02:13:19.203665 30001 raft_consensus.cc:2454] T 0a9b0de9127e46d59a07db2566a020ce P aab09611a057443b8fe9647d89e2216d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e446651a93924539908dd74094343d20 in term 1.
I20260926 02:13:19.204048 30201 leader_election.cc:302] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aab09611a057443b8fe9647d89e2216d, e446651a93924539908dd74094343d20; no voters:
I20260926 02:13:19.204341 30338 raft_consensus.cc:2781] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:19.204811 30338 raft_consensus.cc:686] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [term 1 LEADER]: Becoming Leader. State: Replica: e446651a93924539908dd74094343d20, State: Running, Role: LEADER
I20260926 02:13:19.205732 30338 consensus_queue.cc:237] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab09611a057443b8fe9647d89e2216d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42499 } } peers { permanent_uuid: "e446651a93924539908dd74094343d20" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35151 } } peers { permanent_uuid: "bbe4da2bd3044d4294446c8101382d70" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45485 } }
I20260926 02:13:19.210508 29855 catalog_manager.cc:5495] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 reported cstate change: term changed from 0 to 1, leader changed from <none> to e446651a93924539908dd74094343d20 (127.28.152.131). New cstate: current_term: 1 leader_uuid: "e446651a93924539908dd74094343d20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab09611a057443b8fe9647d89e2216d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e446651a93924539908dd74094343d20" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbe4da2bd3044d4294446c8101382d70" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45485 } health_report { overall_health: UNKNOWN } } }
Sep 26 02:13:19 dist-test-slave-hllj krb5kdc[29812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:13:19.311537 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.296036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.28.152.129:42499)
0926 02:13:19.296258 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:19.296266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:19.296704 (+ 438us) server_negotiation.cc:407] Connection header received
0926 02:13:19.296853 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:19.296857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:19.296920 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:19.297013 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:19.298865 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.299667 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.301071 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.301546 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.301643 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:19.305700 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:19.305718 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:19.305735 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:19.305781 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:19.308454 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.309316 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.309325 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.309327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.309547 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.309953 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.309957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.309959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.310392 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:19.310623 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:19.310993 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 02:13:19.311169 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 02:13:19 dist-test-slave-hllj krb5kdc[29812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388799, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:13:19 dist-test-slave-hllj krb5kdc[29812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388799, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:13:19.365479 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.347512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.28.152.129:42499)
0926 02:13:19.347723 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:19.347729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:19.348021 (+ 292us) server_negotiation.cc:407] Connection header received
0926 02:13:19.348250 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:19.348256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:19.348333 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:19.348417 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:19.349801 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.350541 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.351804 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.352236 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.352345 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:19.360351 (+ 8006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:19.360356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:19.360360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:19.360401 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:19.362820 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.363542 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.363546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.363549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.363678 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.364058 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.364062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.364064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.364337 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:19.364563 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:19.364875 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 02:13:19.365098 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 02:13:19 dist-test-slave-hllj krb5kdc[29812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:13:19 dist-test-slave-hllj krb5kdc[29812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:13:19.409610 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.394856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.28.152.129:42499)
0926 02:13:19.395032 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:19.395040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:19.395385 (+ 345us) server_negotiation.cc:407] Connection header received
0926 02:13:19.395839 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:19.395843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:19.395911 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:19.396000 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:19.397376 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.398081 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.399379 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.399830 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.399943 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:19.404277 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:19.404283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:19.404286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:19.404328 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:19.406875 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.407623 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.407627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.407630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.407769 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.408141 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.408145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.408147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.408436 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:19.408673 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:19.408994 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 02:13:19.409257 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 02:13:19.411300 29961 tablet_service.cc:2908] Rejecting scan request for tablet 0a9b0de9127e46d59a07db2566a020ce: Uninitialized: clean time has not yet been initialized
I20260926 02:13:19.414614 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 29918
I20260926 02:13:19.428351 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.415690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.28.152.129:42499)
0926 02:13:19.415854 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:19.415860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:19.416195 (+ 335us) server_negotiation.cc:407] Connection header received
0926 02:13:19.416355 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:19.416358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:19.416422 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:19.416505 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:19.417744 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.418448 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.419606 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.420008 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.420113 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:19.422402 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:19.422408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:19.422411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:19.422447 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:19.425520 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.426313 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.426318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.426320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.426453 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.426950 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.426955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.426957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.427335 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:19.427543 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:19.427930 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 02:13:19.428121 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 02:13:19.429121 30041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:19.516016 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 29918
I20260926 02:13:19.529376 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 30052
Sep 26 02:13:19 dist-test-slave-hllj krb5kdc[29812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:13:19.544391 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.530136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.28.152.130:45485)
0926 02:13:19.530319 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:19.530324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:19.530345 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:13:19.530516 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:19.530521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:19.530584 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:19.530681 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:19.531862 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.532583 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.533954 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.534341 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.534460 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:19.538641 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:19.538659 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:19.538675 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:19.538715 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:19.541387 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.542161 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.542170 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.542172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.542318 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.542705 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.542709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.542711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.543105 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:19.543392 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:19.543834 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 02:13:19.544060 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:13:19.545071 30175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:19.641477 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 30052
W20260926 02:13:19.654865 30202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.152.130:45485 (error 108)
I20260926 02:13:19.655025 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 30185
I20260926 02:13:19.656770 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.656308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:45485 (local address 127.28.152.131:47347)
0926 02:13:19.656452 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:19.656560 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.130:45485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 02:13:19.657070 30202 consensus_peers.cc:487] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 -> Peer bbe4da2bd3044d4294446c8101382d70 (127.28.152.130:45485): Couldn't send request to peer bbe4da2bd3044d4294446c8101382d70. Status: Network error: Client connection negotiation failed: client connection to 127.28.152.130:45485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 02:13:19.657655 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.657246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42499 (local address 127.28.152.131:53969)
0926 02:13:19.657446 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:19.657526 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 02:13:19.658569 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.658196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42499 (local address 127.28.152.131:35095)
0926 02:13:19.658356 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:19.658428 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 02:13:19.658735 30201 consensus_peers.cc:487] T 0a9b0de9127e46d59a07db2566a020ce P e446651a93924539908dd74094343d20 -> Peer aab09611a057443b8fe9647d89e2216d (127.28.152.129:42499): Couldn't send request to peer aab09611a057443b8fe9647d89e2216d. Status: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 26 02:13:19 dist-test-slave-hllj krb5kdc[29812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:13:19.670143 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:19.655791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.28.152.131:35151)
0926 02:13:19.656213 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:19.656217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:19.656321 (+ 104us) server_negotiation.cc:407] Connection header received
0926 02:13:19.656556 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:19.656562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:19.656637 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:19.656761 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:19.658350 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.659178 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.660481 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:19.660880 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:19.660978 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:19.664877 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:19.664891 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:19.664912 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:19.664949 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:19.667625 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.668265 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.668272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.668274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.668458 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:19.668762 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:19.668766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:19.668768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:19.669115 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:19.669439 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:19.669622 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 02:13:19.669788 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":204,"threads_started":1}
I20260926 02:13:19.670784 30308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:19.759018 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 30185
I20260926 02:13:19.772410 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 29825
I20260926 02:13:19.773613 29887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:19.878315 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 29825
2026-09-26T02:13:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4325 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:13:19 dist-test-slave-hllj krb5kdc[30362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:13:19 dist-test-slave-hllj krb5kdc[30362](info): set up 2 sockets
Sep 26 02:13:19 dist-test-slave-hllj krb5kdc[30362](info): commencing operation
krb5kdc: starting...
W20260926 02:13:20.878525 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:13:20 dist-test-slave-hllj krb5kdc[30362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:13:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:13:21.060401 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:39735
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:40737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:39735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:21.383059 30375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:21.383365 30375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:21.383490 30375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:21.390913 30375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:21.391036 30375 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:13:21.391083 30375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:21.391163 30375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:21.391227 30375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:21.391310 30375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:13:21.401489 30375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:39735
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:39735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:21.402787 30375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:21.404175 30375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:21.416169 30380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:21.416090 30381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:21.416271 30383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:21.417080 30375 server_base.cc:1034] running on GCE node
I20260926 02:13:21.418036 30375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:21.419569 30375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:21.420832 30375 hybrid_clock.cc:648] HybridClock initialized: now 1790388801420760 us; error 57 us; skew 500 ppm
I20260926 02:13:21.421247 30375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:21 dist-test-slave-hllj krb5kdc[30362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388801, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:21.424599 30375 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:13:21.426246 30375 webserver.cc:460] Webserver started at http://127.28.152.190:33487/ using document root <none> and password file <none>
I20260926 02:13:21.426981 30375 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:21.427085 30375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:21.427405 30375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:21.430500 30375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "609c0e5c8fe642579c485e7684626764"
format_stamp: "Formatted at 2026-09-26 02:13:21 on dist-test-slave-hllj"
server_key: "2feb41878a73abd4349b98bab01f9a24"
server_key_iv: "d7e5217354d25a6508c154c618a68a34"
server_key_version: "encryptionkey@0"
I20260926 02:13:21.431296 30375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "609c0e5c8fe642579c485e7684626764"
format_stamp: "Formatted at 2026-09-26 02:13:21 on dist-test-slave-hllj"
server_key: "2feb41878a73abd4349b98bab01f9a24"
server_key_iv: "d7e5217354d25a6508c154c618a68a34"
server_key_version: "encryptionkey@0"
I20260926 02:13:21.437063 30375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:13:21.441058 30390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:21.442639 30375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:13:21.442834 30375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "609c0e5c8fe642579c485e7684626764"
format_stamp: "Formatted at 2026-09-26 02:13:21 on dist-test-slave-hllj"
server_key: "2feb41878a73abd4349b98bab01f9a24"
server_key_iv: "d7e5217354d25a6508c154c618a68a34"
server_key_version: "encryptionkey@0"
I20260926 02:13:21.443063 30375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:21.459645 30375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:21.464170 30375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:21.464444 30375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:21.485791 30375 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:39735
I20260926 02:13:21.485800 30451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:39735 every 8 connection(s)
I20260926 02:13:21.487259 30375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:21.487840 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 30375
I20260926 02:13:21.488073 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:21.488862 29282 external_mini_cluster.cc:1428] Setting key 05c16bada05981fe1eb1b2909a35b00e
I20260926 02:13:21.491854 30452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:13:21 dist-test-slave-hllj krb5kdc[30362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:21.505210 30452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764: Bootstrap starting.
I20260926 02:13:21.508729 30452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:21.510069 30452 log.cc:824] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:21.512866 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:21.491101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37548 (local address 127.28.152.190:39735)
0926 02:13:21.491868 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:21.491880 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:21.491938 (+ 58us) server_negotiation.cc:407] Connection header received
0926 02:13:21.493639 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:21.493663 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:21.494082 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:21.494442 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:21.496241 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:21.497722 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:21.499496 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:21.500207 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:21.500344 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:21.505450 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:21.505461 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:21.505479 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:21.505526 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:21.508503 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:21.509412 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:21.509420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:21.509427 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:21.509602 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:21.510038 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:21.510041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:21.510043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:21.510713 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:21.511042 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:21.511404 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 02:13:21.511833 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":339,"threads_started":1}
I20260926 02:13:21.513224 30452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764: No bootstrap required, opened a new log
I20260926 02:13:21.521090 30452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609c0e5c8fe642579c485e7684626764" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39735 } }
I20260926 02:13:21.521528 30452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:21.521618 30452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609c0e5c8fe642579c485e7684626764, State: Initialized, Role: FOLLOWER
I20260926 02:13:21.522213 30452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609c0e5c8fe642579c485e7684626764" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39735 } }
I20260926 02:13:21.522526 30452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:21.522676 30452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:21.522861 30452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:21.524981 30452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609c0e5c8fe642579c485e7684626764" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39735 } }
I20260926 02:13:21.525559 30452 leader_election.cc:302] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 609c0e5c8fe642579c485e7684626764; no voters:
I20260926 02:13:21.526093 30452 leader_election.cc:288] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:21.526219 30457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:21.526901 30457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [term 1 LEADER]: Becoming Leader. State: Replica: 609c0e5c8fe642579c485e7684626764, State: Running, Role: LEADER
I20260926 02:13:21.527557 30457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609c0e5c8fe642579c485e7684626764" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39735 } }
I20260926 02:13:21.528258 30452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:21.530576 30459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 609c0e5c8fe642579c485e7684626764. Latest consensus state: current_term: 1 leader_uuid: "609c0e5c8fe642579c485e7684626764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609c0e5c8fe642579c485e7684626764" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39735 } } }
I20260926 02:13:21.530730 30458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "609c0e5c8fe642579c485e7684626764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609c0e5c8fe642579c485e7684626764" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39735 } } }
I20260926 02:13:21.531068 30458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:21.530969 30459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:21.536705 30462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:21.542191 30462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:21.553341 30462 catalog_manager.cc:1347] Generated new cluster ID: 631bde28775b4694ac4b6c08a037327a
I20260926 02:13:21.553467 30462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:21.578733 30462 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:21.580132 30462 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:21.586762 30462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764: Generated new TSK 0
I20260926 02:13:21.587677 30462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:13:21.633410 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39735
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:40737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:22.026458 30478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:22.026765 30478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:22.026947 30478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:22.034366 30478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:22.034492 30478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:22.034602 30478 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:13:22.034693 30478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:13:22.044988 30478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.152.190:39735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:22.046474 30478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:22.047914 30478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:22.061542 30484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:22.061579 30483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:22.061733 30486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:22.062338 30478 server_base.cc:1034] running on GCE node
I20260926 02:13:22.062961 30478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:22.064002 30478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:22.065258 30478 hybrid_clock.cc:648] HybridClock initialized: now 1790388802065186 us; error 39 us; skew 500 ppm
I20260926 02:13:22.065595 30478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:22 dist-test-slave-hllj krb5kdc[30362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:22.068872 30478 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:13:22.070602 30478 webserver.cc:460] Webserver started at http://127.28.152.129:38243/ using document root <none> and password file <none>
I20260926 02:13:22.071317 30478 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:22.071419 30478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:22.071727 30478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:22.074702 30478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "c832b029a22043f1897594644bf1dbd7"
format_stamp: "Formatted at 2026-09-26 02:13:22 on dist-test-slave-hllj"
server_key: "7f31d8e3dabc2b93fd40ab3a37d0dbf9"
server_key_iv: "c0689b89cf3adebca0603c6b4f87a378"
server_key_version: "encryptionkey@0"
I20260926 02:13:22.075479 30478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "c832b029a22043f1897594644bf1dbd7"
format_stamp: "Formatted at 2026-09-26 02:13:22 on dist-test-slave-hllj"
server_key: "7f31d8e3dabc2b93fd40ab3a37d0dbf9"
server_key_iv: "c0689b89cf3adebca0603c6b4f87a378"
server_key_version: "encryptionkey@0"
I20260926 02:13:22.080735 30478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 02:13:22.084352 30493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:22.085899 30478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:13:22.086066 30478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "c832b029a22043f1897594644bf1dbd7"
format_stamp: "Formatted at 2026-09-26 02:13:22 on dist-test-slave-hllj"
server_key: "7f31d8e3dabc2b93fd40ab3a37d0dbf9"
server_key_iv: "c0689b89cf3adebca0603c6b4f87a378"
server_key_version: "encryptionkey@0"
I20260926 02:13:22.086272 30478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:22.101603 30478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:22.106249 30478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:22.106521 30478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:22.109371 30478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:22.109488 30478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:22.109615 30478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:22.109690 30478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:13:22 dist-test-slave-hllj krb5kdc[30362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:22.133041 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:22.109836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:55355 (local address 127.28.152.190:39735)
0926 02:13:22.110552 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:22.110557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:22.111639 (+ 1082us) server_negotiation.cc:407] Connection header received
0926 02:13:22.113678 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:22.113684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:22.113781 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:22.113875 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:22.116250 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.117227 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.120346 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.121012 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.121189 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:22.126256 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:22.126261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:22.126264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:22.126300 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:22.129815 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:22.130580 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:22.130586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:22.130591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:22.130754 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:22.131152 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:22.131157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:22.131161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:22.131513 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:22.131656 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:22.132358 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 02:13:22.132604 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":532,"threads_started":1}
I20260926 02:13:22.133961 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:22.110616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39735 (local address 127.28.152.129:55355)
0926 02:13:22.111471 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:22.111520 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:22.113390 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:22.114193 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:22.114205 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:22.115066 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:22.116044 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:22.116076 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.117451 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.117457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:22.120078 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:22.120101 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.121316 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.121327 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:22.121607 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:22.122421 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:22.122455 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:22.125938 (+ 3483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:22.129985 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:22.129996 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:22.130016 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:22.130463 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:22.130849 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:22.130852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:22.130854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:22.131060 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:22.131640 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:22.131646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:22.132004 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 02:13:22.132238 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 02:13:22.132681 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":365,"threads_started":1}
I20260926 02:13:22.154623 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:22.140195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:56883 (local address 127.28.152.190:39735)
0926 02:13:22.140399 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:22.140406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:22.141601 (+ 1195us) server_negotiation.cc:407] Connection header received
0926 02:13:22.141823 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:22.141828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:22.141906 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:22.142006 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:22.142987 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.143748 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.145037 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.145576 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.145705 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:22.148196 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:22.148201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:22.148205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:22.148237 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:22.151482 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:22.152432 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:22.152437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:22.152440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:22.152593 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:22.153074 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:22.153080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:22.153083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:22.153511 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:22.153653 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:22.154197 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 02:13:22.154386 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:13:22.154939 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:22.140156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39735 (local address 127.28.152.129:56883)
0926 02:13:22.141430 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:22.141461 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:22.141681 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:22.142064 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:22.142068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:22.142559 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:22.142843 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:22.142853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.143878 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.143884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:22.144886 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:22.144898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.145820 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.145828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:22.146048 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:22.146815 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:22.146841 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:22.147998 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:22.151662 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:22.151671 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:22.151675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:22.152285 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:22.152727 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:22.152733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:22.152736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:22.152927 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:22.153667 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:22.153673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:22.153899 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 02:13:22.154076 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 02:13:22.154467 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":627,"threads_started":1}
I20260926 02:13:22.162878 30478 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:40269
I20260926 02:13:22.162909 30613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:40269 every 8 connection(s)
I20260926 02:13:22.164038 30478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:13:22.166123 30614 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39735
I20260926 02:13:22.166355 30614 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:22.167043 30614 heartbeater.cc:511] Master 127.28.152.190:39735 requested a full tablet report, sending...
I20260926 02:13:22.169028 30407 ts_manager.cc:194] Registered new tserver with Master: c832b029a22043f1897594644bf1dbd7 (127.28.152.129:40269)
I20260926 02:13:22.170672 30407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:55355
I20260926 02:13:22.174050 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 30478
I20260926 02:13:22.174233 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:13:22.174839 29282 external_mini_cluster.cc:1428] Setting key 551bf2c9f09601b9d76a81101dfaf1d3
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:13:22.212761 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39735
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:40737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:22.534252 30618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:22.534567 30618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:22.534698 30618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:22.542104 30618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:22.542232 30618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:22.542356 30618 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:13:22.542474 30618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:13:22.552575 30618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.152.190:39735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:22.553944 30618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:22.555362 30618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:22.568614 30623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:22.568665 30624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:22.568804 30626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:22.569416 30618 server_base.cc:1034] running on GCE node
I20260926 02:13:22.570024 30618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:22.571028 30618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:22.572247 30618 hybrid_clock.cc:648] HybridClock initialized: now 1790388802572157 us; error 69 us; skew 500 ppm
I20260926 02:13:22.572656 30618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:22 dist-test-slave-hllj krb5kdc[30362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:22.576170 30618 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:13:22.577818 30618 webserver.cc:460] Webserver started at http://127.28.152.130:33197/ using document root <none> and password file <none>
I20260926 02:13:22.578605 30618 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:22.578706 30618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:22.578979 30618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:22.582175 30618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "d339244f27434bb7ba9377e5606a887b"
format_stamp: "Formatted at 2026-09-26 02:13:22 on dist-test-slave-hllj"
server_key: "b0c430ef46218e640420ca86ea3ffa1e"
server_key_iv: "998edf805e67930f46db7ad3e819d13c"
server_key_version: "encryptionkey@0"
I20260926 02:13:22.583045 30618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "d339244f27434bb7ba9377e5606a887b"
format_stamp: "Formatted at 2026-09-26 02:13:22 on dist-test-slave-hllj"
server_key: "b0c430ef46218e640420ca86ea3ffa1e"
server_key_iv: "998edf805e67930f46db7ad3e819d13c"
server_key_version: "encryptionkey@0"
I20260926 02:13:22.588687 30618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:13:22.592537 30633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:22.594220 30618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:13:22.594452 30618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "d339244f27434bb7ba9377e5606a887b"
format_stamp: "Formatted at 2026-09-26 02:13:22 on dist-test-slave-hllj"
server_key: "b0c430ef46218e640420ca86ea3ffa1e"
server_key_iv: "998edf805e67930f46db7ad3e819d13c"
server_key_version: "encryptionkey@0"
I20260926 02:13:22.594657 30618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:22.615891 30618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:22.620513 30618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:22.620762 30618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:22.623631 30618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:22.623739 30618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:22.623900 30618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:22.623980 30618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:13:22 dist-test-slave-hllj krb5kdc[30362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388802, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:22.647233 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:22.624176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:52993 (local address 127.28.152.190:39735)
0926 02:13:22.624314 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:22.624321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:22.626011 (+ 1690us) server_negotiation.cc:407] Connection header received
0926 02:13:22.628601 (+ 2590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:22.628607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:22.628698 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:22.628794 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:22.630702 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.631421 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.633163 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.633705 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.633837 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:22.638883 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:22.638888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:22.638891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:22.638927 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:22.641445 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:22.642257 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:22.642263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:22.642267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:22.642442 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:22.642990 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:22.642996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:22.642999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:22.643373 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:22.643557 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:22.646696 (+ 3139us) server_negotiation.cc:299] Negotiation successful
0926 02:13:22.646941 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 02:13:22.647799 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:22.624912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39735 (local address 127.28.152.130:52993)
0926 02:13:22.625818 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:22.625861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:22.628281 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:22.629153 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:22.629167 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:22.629842 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:22.630506 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:22.630526 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.631584 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.631588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:22.632973 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:22.632987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.634200 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.634210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:22.634626 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:22.635514 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:22.635560 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:22.638641 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:22.641598 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:22.641609 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:22.641630 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:22.642111 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:22.642581 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:22.642585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:22.642587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:22.642835 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:22.643529 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:22.643542 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:22.644041 (+ 499us) client_negotiation.cc:769] Sending connection context
0926 02:13:22.644415 (+ 374us) client_negotiation.cc:240] Negotiation successful
0926 02:13:22.644996 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":409,"threads_started":1}
I20260926 02:13:22.667845 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:22.654535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:56937 (local address 127.28.152.190:39735)
0926 02:13:22.654734 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:22.654740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:22.655539 (+ 799us) server_negotiation.cc:407] Connection header received
0926 02:13:22.655681 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:22.655685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:22.655752 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:22.655838 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:22.657114 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.658089 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.659849 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.660241 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.660355 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:22.662326 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:22.662331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:22.662335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:22.662377 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:22.664859 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:22.665791 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:22.665797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:22.665800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:22.665963 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:22.666329 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:22.666333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:22.666335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:22.666689 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:22.666827 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:22.667254 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 02:13:22.667483 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:13:22.667935 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:22.654617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39735 (local address 127.28.152.130:56937)
0926 02:13:22.655344 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:22.655374 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:22.655552 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:22.655892 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:22.655897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:22.656528 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:22.656938 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:22.656951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.658233 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.658238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:22.659708 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:22.659722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:22.660461 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:22.660467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:22.660635 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:22.661403 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:22.661421 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:22.662144 (+ 723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:22.665022 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:22.665028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:22.665032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:22.665668 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:22.666079 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:22.666084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:22.666088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:22.666216 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:22.666820 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:22.666826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:22.667016 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 02:13:22.667137 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 02:13:22.667483 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":373,"threads_started":1}
I20260926 02:13:22.683333 30618 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:35325
I20260926 02:13:22.683349 30752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:35325 every 8 connection(s)
I20260926 02:13:22.684559 30618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:13:22.686610 30753 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39735
I20260926 02:13:22.686810 30753 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:22.687453 30753 heartbeater.cc:511] Master 127.28.152.190:39735 requested a full tablet report, sending...
I20260926 02:13:22.688776 30407 ts_manager.cc:194] Registered new tserver with Master: d339244f27434bb7ba9377e5606a887b (127.28.152.130:35325)
I20260926 02:13:22.689637 30407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:52993
I20260926 02:13:22.690559 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 30618
I20260926 02:13:22.690722 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:13:22.691098 29282 external_mini_cluster.cc:1428] Setting key 9aee1ac56c0ba44e2e0ae0acc015d034
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:13:22.732779 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39735
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:40737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:23.060784 30757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:23.061072 30757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:23.061215 30757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:23.068650 30757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:23.068768 30757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:23.068851 30757 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:13:23.068930 30757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:13:23.078966 30757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.152.190:39735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:23.080215 30757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:23.081547 30757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:23.094529 30762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:23.094609 30763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:23.094556 30765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:23.095309 30757 server_base.cc:1034] running on GCE node
I20260926 02:13:23.095911 30757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:23.096881 30757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:23.098093 30757 hybrid_clock.cc:648] HybridClock initialized: now 1790388803098007 us; error 63 us; skew 500 ppm
I20260926 02:13:23.098405 30757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:23 dist-test-slave-hllj krb5kdc[30362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388803, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:23.101485 30757 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:13:23.102959 30757 webserver.cc:460] Webserver started at http://127.28.152.131:33247/ using document root <none> and password file <none>
I20260926 02:13:23.103637 30757 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:23.103724 30757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:23.103991 30757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:23.106930 30757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "286bd6ee6c7a4aaeb09d731f34156bd8"
format_stamp: "Formatted at 2026-09-26 02:13:23 on dist-test-slave-hllj"
server_key: "3e26c03d157cd0eea522336eaec510dc"
server_key_iv: "4ae5703f0a95a9da4145087f77ea5caa"
server_key_version: "encryptionkey@0"
I20260926 02:13:23.107679 30757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "286bd6ee6c7a4aaeb09d731f34156bd8"
format_stamp: "Formatted at 2026-09-26 02:13:23 on dist-test-slave-hllj"
server_key: "3e26c03d157cd0eea522336eaec510dc"
server_key_iv: "4ae5703f0a95a9da4145087f77ea5caa"
server_key_version: "encryptionkey@0"
I20260926 02:13:23.112778 30757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:13:23.116261 30772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:23.117779 30757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:13:23.117941 30757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "286bd6ee6c7a4aaeb09d731f34156bd8"
format_stamp: "Formatted at 2026-09-26 02:13:23 on dist-test-slave-hllj"
server_key: "3e26c03d157cd0eea522336eaec510dc"
server_key_iv: "4ae5703f0a95a9da4145087f77ea5caa"
server_key_version: "encryptionkey@0"
I20260926 02:13:23.118163 30757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:23.137029 30757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:23.141409 30757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:23.141641 30757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:23.144405 30757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:23.144518 30757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:23.144654 30757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:23.144721 30757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:13:23 dist-test-slave-hllj krb5kdc[30362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388803, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:23.166940 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.145087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:39937 (local address 127.28.152.190:39735)
0926 02:13:23.145263 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.145270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.147065 (+ 1795us) server_negotiation.cc:407] Connection header received
0926 02:13:23.149365 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.149372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.149469 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.149597 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:23.151873 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.153054 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.154797 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.155444 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.155747 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.160679 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:23.160688 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:23.160693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:23.160741 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:23.163694 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:23.164543 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:23.164549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:23.164552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:23.164686 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:23.165104 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:23.165110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:23.165113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:23.165450 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:23.165582 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.166356 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.166588 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:13:23.168051 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.145952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39735 (local address 127.28.152.131:39937)
0926 02:13:23.146897 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.146939 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.149007 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.149859 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.149867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.150667 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:23.151672 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.151697 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.153255 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.153259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.154636 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.154650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.155776 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.155783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.156061 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.156884 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:23.156909 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:23.160344 (+ 3435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:23.163884 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:23.163895 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:23.163914 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:23.164418 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:23.164792 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:23.164797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:23.164801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:23.164997 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:23.165573 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:23.165582 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:23.165978 (+ 396us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.166219 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.166771 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":364,"threads_started":1}
I20260926 02:13:23.173552 30614 heartbeater.cc:503] Master 127.28.152.190:39735 was elected leader, sending a full tablet report...
I20260926 02:13:23.186471 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.173286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:51093 (local address 127.28.152.190:39735)
0926 02:13:23.173487 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.173493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.174035 (+ 542us) server_negotiation.cc:407] Connection header received
0926 02:13:23.174186 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.174191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.174276 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.174352 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:23.175289 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.176050 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.177349 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.177914 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.178032 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.180620 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:23.180628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:23.180632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:23.180672 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:23.183577 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:23.184436 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:23.184439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:23.184442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:23.184595 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:23.185001 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:23.185005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:23.185009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:23.185397 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:23.185540 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.185905 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.186116 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:13:23.186592 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.173354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39735 (local address 127.28.152.131:51093)
0926 02:13:23.173945 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.173960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.174080 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.174400 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.174404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.174807 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:23.175131 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.175143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.176195 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.176199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.177230 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.177242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.178270 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.178280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.178500 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.179271 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:23.179288 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:23.180437 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:23.183742 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:23.183748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:23.183752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:23.184306 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:23.184732 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:23.184740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:23.184743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:23.184876 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:23.185511 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:23.185516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:23.185690 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.185805 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.186164 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":306,"threads_started":1}
I20260926 02:13:23.200574 30757 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:42667
I20260926 02:13:23.200593 30891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:42667 every 8 connection(s)
I20260926 02:13:23.201838 30757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:13:23.203871 30892 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39735
I20260926 02:13:23.204105 30892 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:23.204741 30892 heartbeater.cc:511] Master 127.28.152.190:39735 requested a full tablet report, sending...
I20260926 02:13:23.206161 30407 ts_manager.cc:194] Registered new tserver with Master: 286bd6ee6c7a4aaeb09d731f34156bd8 (127.28.152.131:42667)
I20260926 02:13:23.206933 30407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:39937
I20260926 02:13:23.211036 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 30757
I20260926 02:13:23.211189 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:13:23.211619 29282 external_mini_cluster.cc:1428] Setting key 140cea173f56fac48f08194484ef3af6
I20260926 02:13:23.214771 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:13:23 dist-test-slave-hllj krb5kdc[30362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:13:23 dist-test-slave-hllj krb5kdc[30362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:23.257196 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.243811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.28.152.190:39735)
0926 02:13:23.243977 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.243982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.244332 (+ 350us) server_negotiation.cc:407] Connection header received
0926 02:13:23.244474 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.244478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.244570 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.244660 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:23.246137 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.246981 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.248297 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.248723 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.248830 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.252447 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:23.252452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:23.252455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:23.252494 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:23.254916 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:23.255478 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:23.255481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:23.255483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:23.255603 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:23.255941 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:23.255944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:23.255946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:23.256251 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:23.256429 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.256686 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.256861 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:13:23.261317 30407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:13:23.267907 30407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:13:23.288205 30905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.279649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:35325 (local address 127.0.0.1:40512)
0926 02:13:23.280376 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.280419 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.280587 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.281541 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.281549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.281579 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:23.281945 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.281968 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.283695 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.283701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.285856 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.285869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.287235 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.287242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.287437 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.287495 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.287617 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.287802 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":2,"thread_start_us":337,"threads_started":1}
I20260926 02:13:23.289357 30907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.280669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:40269 (local address 127.0.0.1:55074)
0926 02:13:23.281186 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.281202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.281309 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.282352 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.282356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.282383 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:23.282855 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.282867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.284608 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.284613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.286840 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.286854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.288585 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.288591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.288745 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.288794 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.288850 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.288964 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":44,"thread_start_us":243,"threads_started":1}
I20260926 02:13:23.290637 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.281791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:42667 (local address 127.0.0.1:48602)
0926 02:13:23.282563 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.282582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.282779 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.283590 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.283594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.283618 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:23.283971 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.283979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.285797 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.285806 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.287725 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.287737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.289912 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.289919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.290077 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.290099 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.290174 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.290288 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":1,"thread_start_us":491,"threads_started":1}
I20260926 02:13:23.290643 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.280944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.28.152.129:40269)
0926 02:13:23.281763 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.281771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.281798 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:13:23.281889 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.281898 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.282136 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.283030 (+ 894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:23.283230 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.284461 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.287002 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.288369 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.289944 (+ 1575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.290103 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.290163 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.290319 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":548,"threads_started":1}
I20260926 02:13:23.290685 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.281546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.28.152.131:42667)
0926 02:13:23.282128 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.282137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.282677 (+ 540us) server_negotiation.cc:407] Connection header received
0926 02:13:23.282947 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.282954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.283158 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.283308 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:23.284149 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.285624 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.287868 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.289361 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.289747 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.289926 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.290231 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.290382 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"spinlock_wait_cycles":21376,"thread_start_us":397,"threads_started":1}
I20260926 02:13:23.293236 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.279829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.28.152.130:35325)
0926 02:13:23.280330 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.280339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.280513 (+ 174us) server_negotiation.cc:407] Connection header received
0926 02:13:23.280778 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.280786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.281055 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.281286 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:23.282156 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.283567 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.286050 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.287052 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.287208 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.287385 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.287694 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.287945 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":233,"threads_started":1}
I20260926 02:13:23.294327 30544 tablet_service.cc:1467] Processing CreateTablet for tablet 477e6744d2c04ed5a89bda8015ca986e (DEFAULT_TABLE table=test-table [id=23addfbb8e8246cb97a6ea478019010c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:23.295506 30544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477e6744d2c04ed5a89bda8015ca986e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:23.295657 30822 tablet_service.cc:1467] Processing CreateTablet for tablet 477e6744d2c04ed5a89bda8015ca986e (DEFAULT_TABLE table=test-table [id=23addfbb8e8246cb97a6ea478019010c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:23.296474 30683 tablet_service.cc:1467] Processing CreateTablet for tablet 477e6744d2c04ed5a89bda8015ca986e (DEFAULT_TABLE table=test-table [id=23addfbb8e8246cb97a6ea478019010c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:23.297227 30822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477e6744d2c04ed5a89bda8015ca986e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:23.297833 30683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477e6744d2c04ed5a89bda8015ca986e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:23.306959 30911 tablet_bootstrap.cc:492] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: Bootstrap starting.
I20260926 02:13:23.310097 30911 tablet_bootstrap.cc:654] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:23.311160 30911 log.cc:824] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:23.312059 30913 tablet_bootstrap.cc:492] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: Bootstrap starting.
I20260926 02:13:23.312726 30912 tablet_bootstrap.cc:492] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: Bootstrap starting.
I20260926 02:13:23.315745 30912 tablet_bootstrap.cc:654] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:23.316813 30912 log.cc:824] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:23.316915 30913 tablet_bootstrap.cc:654] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:23.318156 30911 tablet_bootstrap.cc:492] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: No bootstrap required, opened a new log
I20260926 02:13:23.318210 30913 log.cc:824] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:23.318464 30911 ts_tablet_manager.cc:1397] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260926 02:13:23.319535 30912 tablet_bootstrap.cc:492] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: No bootstrap required, opened a new log
I20260926 02:13:23.319777 30912 ts_tablet_manager.cc:1397] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260926 02:13:23.321295 30913 tablet_bootstrap.cc:492] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: No bootstrap required, opened a new log
I20260926 02:13:23.321555 30913 ts_tablet_manager.cc:1397] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:13:23.327666 30912 raft_consensus.cc:348] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.328084 30912 raft_consensus.cc:374] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:23.328179 30912 raft_consensus.cc:729] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d339244f27434bb7ba9377e5606a887b, State: Initialized, Role: FOLLOWER
I20260926 02:13:23.328842 30912 consensus_queue.cc:260] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.329527 30911 raft_consensus.cc:348] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.329977 30911 raft_consensus.cc:374] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:23.330068 30911 raft_consensus.cc:729] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c832b029a22043f1897594644bf1dbd7, State: Initialized, Role: FOLLOWER
I20260926 02:13:23.330271 30753 heartbeater.cc:503] Master 127.28.152.190:39735 was elected leader, sending a full tablet report...
I20260926 02:13:23.330842 30912 ts_tablet_manager.cc:1428] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260926 02:13:23.330804 30911 consensus_queue.cc:260] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.332927 30911 ts_tablet_manager.cc:1428] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 02:13:23.332661 30913 raft_consensus.cc:348] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.333103 30913 raft_consensus.cc:374] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:23.333243 30913 raft_consensus.cc:729] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286bd6ee6c7a4aaeb09d731f34156bd8, State: Initialized, Role: FOLLOWER
I20260926 02:13:23.334596 30913 consensus_queue.cc:260] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.335695 30892 heartbeater.cc:503] Master 127.28.152.190:39735 was elected leader, sending a full tablet report...
I20260926 02:13:23.336376 30913 ts_tablet_manager.cc:1428] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260926 02:13:23.416292 30615 tablet.cc:2369] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:13:23.436705 30754 tablet.cc:2369] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:13:23.453490 30893 tablet.cc:2369] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:13:23.484512 30917 raft_consensus.cc:482] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:23.484845 30917 raft_consensus.cc:504] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.486608 30917 leader_election.cc:288] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c832b029a22043f1897594644bf1dbd7 (127.28.152.129:40269), 286bd6ee6c7a4aaeb09d731f34156bd8 (127.28.152.131:42667)
I20260926 02:13:23.493945 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.487084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:49457 (local address 127.28.152.129:40269)
0926 02:13:23.487290 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.487297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.487879 (+ 582us) server_negotiation.cc:407] Connection header received
0926 02:13:23.487961 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.487966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.488038 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.488120 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:23.488908 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.490046 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.491988 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.492933 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.493106 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.493213 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.493552 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.493693 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:13:23.494006 30920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.486948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:40269 (local address 127.28.152.130:49457)
0926 02:13:23.487561 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.487576 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.487735 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.488292 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.488298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.488326 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:23.488747 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.488759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.490191 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.490197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.491830 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.491842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.493179 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.493185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.493358 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.493397 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.493504 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.493656 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":332,"threads_started":1}
I20260926 02:13:23.495173 30921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.487723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:42667 (local address 127.28.152.130:56363)
0926 02:13:23.488637 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.488655 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.488784 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.489161 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.489165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.489188 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:23.489485 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.489494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.490659 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.490668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.492841 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.492857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.494483 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.494488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.494610 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.494634 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.494715 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.494843 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":561,"threads_started":1}
I20260926 02:13:23.495317 30564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477e6744d2c04ed5a89bda8015ca986e" candidate_uuid: "d339244f27434bb7ba9377e5606a887b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c832b029a22043f1897594644bf1dbd7" is_pre_election: true
I20260926 02:13:23.495867 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.487957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:56363 (local address 127.28.152.131:42667)
0926 02:13:23.488154 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.488159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.488763 (+ 604us) server_negotiation.cc:407] Connection header received
0926 02:13:23.488968 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.488972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.489032 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.489107 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:23.489635 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.490505 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.493002 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.494117 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.494271 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.494340 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.495492 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.495644 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:13:23.495922 30564 raft_consensus.cc:2454] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d339244f27434bb7ba9377e5606a887b in term 0.
I20260926 02:13:23.496663 30634 leader_election.cc:302] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c832b029a22043f1897594644bf1dbd7, d339244f27434bb7ba9377e5606a887b; no voters:
I20260926 02:13:23.496796 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477e6744d2c04ed5a89bda8015ca986e" candidate_uuid: "d339244f27434bb7ba9377e5606a887b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" is_pre_election: true
I20260926 02:13:23.497112 30917 raft_consensus.cc:2781] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:23.497287 30917 raft_consensus.cc:482] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:23.497318 30847 raft_consensus.cc:2454] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d339244f27434bb7ba9377e5606a887b in term 0.
I20260926 02:13:23.497362 30917 raft_consensus.cc:3037] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:23.500638 30917 raft_consensus.cc:504] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.501374 30917 leader_election.cc:288] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [CANDIDATE]: Term 1 election: Requested vote from peers c832b029a22043f1897594644bf1dbd7 (127.28.152.129:40269), 286bd6ee6c7a4aaeb09d731f34156bd8 (127.28.152.131:42667)
I20260926 02:13:23.501928 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477e6744d2c04ed5a89bda8015ca986e" candidate_uuid: "d339244f27434bb7ba9377e5606a887b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8"
I20260926 02:13:23.501928 30564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477e6744d2c04ed5a89bda8015ca986e" candidate_uuid: "d339244f27434bb7ba9377e5606a887b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c832b029a22043f1897594644bf1dbd7"
I20260926 02:13:23.502153 30847 raft_consensus.cc:3037] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:23.502153 30564 raft_consensus.cc:3037] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:23.505419 30564 raft_consensus.cc:2454] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d339244f27434bb7ba9377e5606a887b in term 1.
I20260926 02:13:23.505419 30847 raft_consensus.cc:2454] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d339244f27434bb7ba9377e5606a887b in term 1.
I20260926 02:13:23.505903 30634 leader_election.cc:302] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286bd6ee6c7a4aaeb09d731f34156bd8, d339244f27434bb7ba9377e5606a887b; no voters:
I20260926 02:13:23.506222 30917 raft_consensus.cc:2781] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:23.506711 30917 raft_consensus.cc:686] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 1 LEADER]: Becoming Leader. State: Replica: d339244f27434bb7ba9377e5606a887b, State: Running, Role: LEADER
I20260926 02:13:23.507229 30917 consensus_queue.cc:237] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.511837 30406 catalog_manager.cc:5495] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b reported cstate change: term changed from 0 to 1, leader changed from <none> to d339244f27434bb7ba9377e5606a887b (127.28.152.130). New cstate: current_term: 1 leader_uuid: "d339244f27434bb7ba9377e5606a887b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } health_report { overall_health: UNKNOWN } } }
I20260926 02:13:23.538827 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.533383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.28.152.130:35325)
0926 02:13:23.533554 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.533558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.533752 (+ 194us) server_negotiation.cc:407] Connection header received
0926 02:13:23.534025 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.534029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.534106 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.534219 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:23.534521 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.535214 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.536599 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.537051 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.537226 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.537759 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:23.538321 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:23.538426 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.538506 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.538575 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 02:13:23.547185 30564 raft_consensus.cc:1261] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7 [term 1 FOLLOWER]: Refusing update from remote peer d339244f27434bb7ba9377e5606a887b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:23.547185 30847 raft_consensus.cc:1261] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 1 FOLLOWER]: Refusing update from remote peer d339244f27434bb7ba9377e5606a887b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:23.548149 30917 consensus_queue.cc:1035] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:23.548894 30922 consensus_queue.cc:1035] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [LEADER]: Connected to new peer: Peer: permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 02:13:23.563256 30926 mvcc.cc:205] Tried to move back new op lower bound from 7333432539317886976 to 7333432539169542144. Current Snapshot: MvccSnapshot[applied={T|T < 7333432539317886976}]
I20260926 02:13:23.581926 30407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:23.582190 30407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:23.586000 30407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764: Sending DeleteTablet for 3 replicas of tablet 477e6744d2c04ed5a89bda8015ca986e
I20260926 02:13:23.587124 30683 tablet_service.cc:1514] Processing DeleteTablet for tablet 477e6744d2c04ed5a89bda8015ca986e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:23 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:40512
I20260926 02:13:23.587255 30544 tablet_service.cc:1514] Processing DeleteTablet for tablet 477e6744d2c04ed5a89bda8015ca986e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:23 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:55074
I20260926 02:13:23.587350 30822 tablet_service.cc:1514] Processing DeleteTablet for tablet 477e6744d2c04ed5a89bda8015ca986e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:23 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:48602
I20260926 02:13:23.589200 30936 tablet_replica.cc:330] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: stopping tablet replica
I20260926 02:13:23.588937 30407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:13:23.589773 30407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:13:23.589937 30935 tablet_replica.cc:330] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: stopping tablet replica
I20260926 02:13:23.590374 30935 raft_consensus.cc:2229] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:13:23.590791 30936 raft_consensus.cc:2229] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:23.591277 30936 raft_consensus.cc:2258] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:23.592216 30935 raft_consensus.cc:2258] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:23.593559 30937 tablet_replica.cc:330] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: stopping tablet replica
I20260926 02:13:23.593992 30937 raft_consensus.cc:2229] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:23.594492 30937 raft_consensus.cc:2258] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:23.594547 30935 ts_tablet_manager.cc:1905] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:23.596833 30937 ts_tablet_manager.cc:1905] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:23.597946 30683 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb9329fd2124e509778834727ab45c0 (DEFAULT_TABLE table=test-table [id=fb58ac51952043c0bc6749e3d3dcc9c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:23.598380 30544 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb9329fd2124e509778834727ab45c0 (DEFAULT_TABLE table=test-table [id=fb58ac51952043c0bc6749e3d3dcc9c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:23.598814 30544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb9329fd2124e509778834727ab45c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:23.599653 30683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb9329fd2124e509778834727ab45c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:23.600255 30936 ts_tablet_manager.cc:1905] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:23.605739 30911 tablet_bootstrap.cc:492] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7: Bootstrap starting.
I20260926 02:13:23.606864 30822 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb9329fd2124e509778834727ab45c0 (DEFAULT_TABLE table=test-table [id=fb58ac51952043c0bc6749e3d3dcc9c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:23.607384 30911 tablet_bootstrap.cc:654] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:23.607524 30912 tablet_bootstrap.cc:492] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b: Bootstrap starting.
I20260926 02:13:23.608023 30936 ts_tablet_manager.cc:1918] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:13:23.608196 30936 log.cc:1196] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/477e6744d2c04ed5a89bda8015ca986e
I20260926 02:13:23.608731 30936 ts_tablet_manager.cc:1939] T 477e6744d2c04ed5a89bda8015ca986e P c832b029a22043f1897594644bf1dbd7: Deleting consensus metadata
I20260926 02:13:23.609215 30912 tablet_bootstrap.cc:654] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:23.614663 30822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb9329fd2124e509778834727ab45c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:23.618314 30911 tablet_bootstrap.cc:492] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7: No bootstrap required, opened a new log
I20260926 02:13:23.618573 30911 ts_tablet_manager.cc:1397] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260926 02:13:23.620164 30911 raft_consensus.cc:348] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.620463 30911 raft_consensus.cc:374] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:23.620594 30911 raft_consensus.cc:729] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c832b029a22043f1897594644bf1dbd7, State: Initialized, Role: FOLLOWER
I20260926 02:13:23.620927 30911 consensus_queue.cc:260] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.621601 30911 ts_tablet_manager.cc:1428] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 02:13:23.622638 30912 tablet_bootstrap.cc:492] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b: No bootstrap required, opened a new log
I20260926 02:13:23.622814 30912 ts_tablet_manager.cc:1397] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20260926 02:13:23.623692 30912 raft_consensus.cc:348] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.623925 30912 raft_consensus.cc:374] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:23.624043 30912 raft_consensus.cc:729] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d339244f27434bb7ba9377e5606a887b, State: Initialized, Role: FOLLOWER
I20260926 02:13:23.624290 30912 consensus_queue.cc:260] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.625358 30912 ts_tablet_manager.cc:1428] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 02:13:23.626317 30937 ts_tablet_manager.cc:1918] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:13:23.626497 30937 log.cc:1196] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/477e6744d2c04ed5a89bda8015ca986e
I20260926 02:13:23.627018 30937 ts_tablet_manager.cc:1939] T 477e6744d2c04ed5a89bda8015ca986e P 286bd6ee6c7a4aaeb09d731f34156bd8: Deleting consensus metadata
I20260926 02:13:23.627153 30935 ts_tablet_manager.cc:1918] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:13:23.627312 30935 log.cc:1196] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/477e6744d2c04ed5a89bda8015ca986e
I20260926 02:13:23.627856 30935 ts_tablet_manager.cc:1939] T 477e6744d2c04ed5a89bda8015ca986e P d339244f27434bb7ba9377e5606a887b: Deleting consensus metadata
I20260926 02:13:23.629813 30391 catalog_manager.cc:4841] TS d339244f27434bb7ba9377e5606a887b (127.28.152.130:35325): tablet 477e6744d2c04ed5a89bda8015ca986e (table test-table [id=23addfbb8e8246cb97a6ea478019010c]) successfully deleted
I20260926 02:13:23.630147 30391 catalog_manager.cc:4841] TS 286bd6ee6c7a4aaeb09d731f34156bd8 (127.28.152.131:42667): tablet 477e6744d2c04ed5a89bda8015ca986e (table test-table [id=23addfbb8e8246cb97a6ea478019010c]) successfully deleted
I20260926 02:13:23.630869 30391 catalog_manager.cc:4841] TS c832b029a22043f1897594644bf1dbd7 (127.28.152.129:40269): tablet 477e6744d2c04ed5a89bda8015ca986e (table test-table [id=23addfbb8e8246cb97a6ea478019010c]) successfully deleted
I20260926 02:13:23.630965 30913 tablet_bootstrap.cc:492] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8: Bootstrap starting.
I20260926 02:13:23.632197 30913 tablet_bootstrap.cc:654] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:23.634065 30913 tablet_bootstrap.cc:492] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8: No bootstrap required, opened a new log
I20260926 02:13:23.634187 30913 ts_tablet_manager.cc:1397] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 02:13:23.634824 30913 raft_consensus.cc:348] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.635051 30913 raft_consensus.cc:374] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:23.635650 30913 raft_consensus.cc:729] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286bd6ee6c7a4aaeb09d731f34156bd8, State: Initialized, Role: FOLLOWER
I20260926 02:13:23.635954 30913 consensus_queue.cc:260] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.636549 30913 ts_tablet_manager.cc:1428] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 02:13:23.643220 30919 raft_consensus.cc:482] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:23.643447 30919 raft_consensus.cc:504] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.645462 30919 leader_election.cc:288] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c832b029a22043f1897594644bf1dbd7 (127.28.152.129:40269), d339244f27434bb7ba9377e5606a887b (127.28.152.130:35325)
I20260926 02:13:23.653393 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.647044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:53323 (local address 127.28.152.130:35325)
0926 02:13:23.647223 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.647227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.647241 (+ 14us) server_negotiation.cc:407] Connection header received
0926 02:13:23.647293 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.647296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.647364 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.647434 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:23.647974 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.648794 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.651601 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.652410 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.652560 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.652634 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.652924 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.653064 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:13:23.653393 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.646372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:35325 (local address 127.28.152.131:53323)
0926 02:13:23.646944 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.646974 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.647078 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.647466 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.647470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.647493 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:23.647828 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.647837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.648944 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.648949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.651416 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.651426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.652611 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.652617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.652778 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.652803 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.652882 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.653018 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":289,"threads_started":1}
I20260926 02:13:23.654649 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb9329fd2124e509778834727ab45c0" candidate_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d339244f27434bb7ba9377e5606a887b" is_pre_election: true
I20260926 02:13:23.655153 30708 raft_consensus.cc:2454] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 286bd6ee6c7a4aaeb09d731f34156bd8 in term 0.
I20260926 02:13:23.655762 30773 leader_election.cc:302] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286bd6ee6c7a4aaeb09d731f34156bd8, d339244f27434bb7ba9377e5606a887b; no voters:
I20260926 02:13:23.655979 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.645998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:49991 (local address 127.28.152.129:40269)
0926 02:13:23.646179 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.646185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.646520 (+ 335us) server_negotiation.cc:407] Connection header received
0926 02:13:23.646597 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.646602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.646676 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.646755 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:23.647580 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.648721 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.651512 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.652794 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.652967 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.653038 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.655557 (+ 2519us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.655706 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:13:23.656235 30919 raft_consensus.cc:2781] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:23.656354 30919 raft_consensus.cc:482] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:23.656437 30919 raft_consensus.cc:3037] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:23.657946 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.646032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:40269 (local address 127.28.152.131:49991)
0926 02:13:23.646235 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.646256 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.646381 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.647186 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.647191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.647219 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:23.647411 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.647423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.648851 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.648857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.651341 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.651362 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.653153 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.653160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.653382 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.653417 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.657192 (+ 3775us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.657329 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":2}
I20260926 02:13:23.658617 30564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb9329fd2124e509778834727ab45c0" candidate_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c832b029a22043f1897594644bf1dbd7" is_pre_election: true
I20260926 02:13:23.658839 30564 raft_consensus.cc:2454] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 286bd6ee6c7a4aaeb09d731f34156bd8 in term 0.
I20260926 02:13:23.660535 30919 raft_consensus.cc:504] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.661263 30919 leader_election.cc:288] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [CANDIDATE]: Term 1 election: Requested vote from peers c832b029a22043f1897594644bf1dbd7 (127.28.152.129:40269), d339244f27434bb7ba9377e5606a887b (127.28.152.130:35325)
I20260926 02:13:23.661736 30564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb9329fd2124e509778834727ab45c0" candidate_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c832b029a22043f1897594644bf1dbd7"
I20260926 02:13:23.661782 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb9329fd2124e509778834727ab45c0" candidate_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d339244f27434bb7ba9377e5606a887b"
I20260926 02:13:23.661942 30564 raft_consensus.cc:3037] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:23.661981 30708 raft_consensus.cc:3037] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:23.664322 30708 raft_consensus.cc:2454] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286bd6ee6c7a4aaeb09d731f34156bd8 in term 1.
I20260926 02:13:23.664882 30564 raft_consensus.cc:2454] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286bd6ee6c7a4aaeb09d731f34156bd8 in term 1.
I20260926 02:13:23.665108 30773 leader_election.cc:302] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286bd6ee6c7a4aaeb09d731f34156bd8, d339244f27434bb7ba9377e5606a887b; no voters:
I20260926 02:13:23.665481 30919 raft_consensus.cc:2781] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:23.665971 30919 raft_consensus.cc:686] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 1 LEADER]: Becoming Leader. State: Replica: 286bd6ee6c7a4aaeb09d731f34156bd8, State: Running, Role: LEADER
I20260926 02:13:23.666375 30919 consensus_queue.cc:237] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.670246 30407 catalog_manager.cc:5495] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 286bd6ee6c7a4aaeb09d731f34156bd8 (127.28.152.131). New cstate: current_term: 1 leader_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } health_report { overall_health: HEALTHY } } }
Sep 26 02:13:23 dist-test-slave-hllj krb5kdc[30362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388801, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:23.727072 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.710740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39735 (local address 127.0.0.1:37578)
0926 02:13:23.711291 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.711309 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.711439 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.711751 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.711754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.712398 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:23.712706 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.712715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.714223 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.714227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.715222 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.715233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.716281 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.716286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.716399 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.717247 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:23.717265 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:23.719874 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:23.724424 (+ 4550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:23.724438 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:23.724445 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:23.724932 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:23.725397 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:23.725404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:23.725406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:23.725543 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:23.726166 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:23.726174 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:23.726357 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.726512 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.726735 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":301,"threads_started":1}
I20260926 02:13:23.727525 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.710703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.28.152.190:39735)
0926 02:13:23.710871 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.710875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.711394 (+ 519us) server_negotiation.cc:407] Connection header received
0926 02:13:23.711541 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.711547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.711626 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.711733 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:23.712858 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.714077 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.715370 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.716096 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.716217 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.720193 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:23.720201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:23.720206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:23.720246 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:23.724240 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:23.725056 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:23.725062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:23.725066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:23.725264 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:23.725639 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:23.725644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:23.725647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:23.726022 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:23.726155 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.726612 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.726912 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":56}
I20260926 02:13:23.732878 30407 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:37578:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:13:23.733989 30407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:13:23.741009 30544 tablet_service.cc:1467] Processing CreateTablet for tablet ba2ebfb8af9448f88dc367be6c6ba56f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a55f6a31d98d4f31bdcc15c1df0e720b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:13:23.741144 30683 tablet_service.cc:1467] Processing CreateTablet for tablet ba2ebfb8af9448f88dc367be6c6ba56f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a55f6a31d98d4f31bdcc15c1df0e720b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:13:23.741333 30822 tablet_service.cc:1467] Processing CreateTablet for tablet ba2ebfb8af9448f88dc367be6c6ba56f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a55f6a31d98d4f31bdcc15c1df0e720b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:13:23.741667 30544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2ebfb8af9448f88dc367be6c6ba56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:23.741683 30683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2ebfb8af9448f88dc367be6c6ba56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:23.741844 30822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2ebfb8af9448f88dc367be6c6ba56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:23.749253 30913 tablet_bootstrap.cc:492] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8: Bootstrap starting.
I20260926 02:13:23.750864 30912 tablet_bootstrap.cc:492] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b: Bootstrap starting.
I20260926 02:13:23.751214 30911 tablet_bootstrap.cc:492] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: Bootstrap starting.
I20260926 02:13:23.751471 30913 tablet_bootstrap.cc:654] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:23.752599 30912 tablet_bootstrap.cc:654] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:23.752737 30911 tablet_bootstrap.cc:654] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:23.761077 30913 tablet_bootstrap.cc:492] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8: No bootstrap required, opened a new log
I20260926 02:13:23.761298 30913 ts_tablet_manager.cc:1397] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20260926 02:13:23.762075 30913 raft_consensus.cc:348] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.762346 30913 raft_consensus.cc:374] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:23.762418 30913 raft_consensus.cc:729] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286bd6ee6c7a4aaeb09d731f34156bd8, State: Initialized, Role: FOLLOWER
I20260926 02:13:23.762688 30913 consensus_queue.cc:260] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.763103 30919 tablet_replica.cc:439] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } } }
I20260926 02:13:23.763383 30919 tablet_replica.cc:442] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:23.764827 30911 tablet_bootstrap.cc:492] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: No bootstrap required, opened a new log
I20260926 02:13:23.765022 30911 ts_tablet_manager.cc:1397] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20260926 02:13:23.765894 30911 raft_consensus.cc:348] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.766122 30912 tablet_bootstrap.cc:492] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b: No bootstrap required, opened a new log
I20260926 02:13:23.766149 30911 raft_consensus.cc:374] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:23.766224 30911 raft_consensus.cc:729] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c832b029a22043f1897594644bf1dbd7, State: Initialized, Role: FOLLOWER
I20260926 02:13:23.766316 30912 ts_tablet_manager.cc:1397] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.010s
I20260926 02:13:23.766325 30913 ts_tablet_manager.cc:1428] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 02:13:23.766507 30911 consensus_queue.cc:260] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.766981 30918 tablet_replica.cc:439] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } } }
I20260926 02:13:23.767365 30918 tablet_replica.cc:442] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:23.767179 30912 raft_consensus.cc:348] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.767503 30912 raft_consensus.cc:374] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:23.767588 30912 raft_consensus.cc:729] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d339244f27434bb7ba9377e5606a887b, State: Initialized, Role: FOLLOWER
I20260926 02:13:23.767882 30912 consensus_queue.cc:260] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.769176 30911 ts_tablet_manager.cc:1428] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:13:23.768908 30932 tablet_replica.cc:439] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } } }
I20260926 02:13:23.769841 30932 tablet_replica.cc:442] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:23.770336 30912 ts_tablet_manager.cc:1428] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260926 02:13:23.784476 30918 raft_consensus.cc:482] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:23.784718 30918 raft_consensus.cc:504] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.786607 30918 leader_election.cc:288] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d339244f27434bb7ba9377e5606a887b (127.28.152.130:35325), 286bd6ee6c7a4aaeb09d731f34156bd8 (127.28.152.131:42667)
I20260926 02:13:23.794880 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.787085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:46371 (local address 127.28.152.130:35325)
0926 02:13:23.787293 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.787299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.788410 (+ 1111us) server_negotiation.cc:407] Connection header received
0926 02:13:23.788512 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.788519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.788630 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.788723 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:23.789742 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.790819 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.792896 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.793813 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.793987 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.794069 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.794401 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.794559 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:13:23.794922 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.786887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:35325 (local address 127.28.152.129:46371)
0926 02:13:23.788088 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.788108 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.788260 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.789064 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.789069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.789097 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:23.789550 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.789564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.790995 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.791001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.792714 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.792727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.794056 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.794062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.794220 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.794254 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.794351 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.794504 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":1,"thread_start_us":897,"threads_started":1}
I20260926 02:13:23.795765 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2ebfb8af9448f88dc367be6c6ba56f" candidate_uuid: "c832b029a22043f1897594644bf1dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d339244f27434bb7ba9377e5606a887b" is_pre_election: true
I20260926 02:13:23.796033 30708 raft_consensus.cc:2454] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c832b029a22043f1897594644bf1dbd7 in term 0.
I20260926 02:13:23.796446 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.788454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:32783 (local address 127.28.152.131:42667)
0926 02:13:23.788638 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.788645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.789280 (+ 635us) server_negotiation.cc:407] Connection header received
0926 02:13:23.789618 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.789623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.789706 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.789779 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:23.790414 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.791543 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.794185 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.795377 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.795522 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.795591 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.796026 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.796173 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:13:23.796525 30494 leader_election.cc:302] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c832b029a22043f1897594644bf1dbd7, d339244f27434bb7ba9377e5606a887b; no voters:
I20260926 02:13:23.797444 30918 raft_consensus.cc:2781] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:23.797606 30918 raft_consensus.cc:482] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:23.797712 30918 raft_consensus.cc:3037] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:23.797834 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.788267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:42667 (local address 127.28.152.129:32783)
0926 02:13:23.789155 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.789174 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.789398 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.789820 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.789824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.789851 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:23.790224 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.790236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.791810 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.791816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.794032 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.794048 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.795625 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.795632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.795812 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.795850 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.796694 (+ 844us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.796855 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"spinlock_wait_cycles":1408,"thread_start_us":526,"threads_started":1}
I20260926 02:13:23.798624 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2ebfb8af9448f88dc367be6c6ba56f" candidate_uuid: "c832b029a22043f1897594644bf1dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" is_pre_election: true
I20260926 02:13:23.798913 30847 raft_consensus.cc:2454] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c832b029a22043f1897594644bf1dbd7 in term 0.
I20260926 02:13:23.800446 30918 raft_consensus.cc:504] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.801044 30918 leader_election.cc:288] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [CANDIDATE]: Term 1 election: Requested vote from peers d339244f27434bb7ba9377e5606a887b (127.28.152.130:35325), 286bd6ee6c7a4aaeb09d731f34156bd8 (127.28.152.131:42667)
I20260926 02:13:23.801673 30847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2ebfb8af9448f88dc367be6c6ba56f" candidate_uuid: "c832b029a22043f1897594644bf1dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8"
I20260926 02:13:23.801662 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2ebfb8af9448f88dc367be6c6ba56f" candidate_uuid: "c832b029a22043f1897594644bf1dbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d339244f27434bb7ba9377e5606a887b"
I20260926 02:13:23.801862 30847 raft_consensus.cc:3037] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:23.801891 30708 raft_consensus.cc:3037] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:23.804723 30847 raft_consensus.cc:2454] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c832b029a22043f1897594644bf1dbd7 in term 1.
I20260926 02:13:23.804920 30708 raft_consensus.cc:2454] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c832b029a22043f1897594644bf1dbd7 in term 1.
I20260926 02:13:23.805226 30494 leader_election.cc:302] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286bd6ee6c7a4aaeb09d731f34156bd8, c832b029a22043f1897594644bf1dbd7; no voters:
I20260926 02:13:23.805537 30918 raft_consensus.cc:2781] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:23.806028 30918 raft_consensus.cc:686] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [term 1 LEADER]: Becoming Leader. State: Replica: c832b029a22043f1897594644bf1dbd7, State: Running, Role: LEADER
I20260926 02:13:23.806463 30918 consensus_queue.cc:237] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } }
I20260926 02:13:23.809162 30955 tablet_replica.cc:439] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: TxnStatusTablet state changed. Reason: New leader c832b029a22043f1897594644bf1dbd7. Latest consensus state: current_term: 1 leader_uuid: "c832b029a22043f1897594644bf1dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } } }
I20260926 02:13:23.809543 30955 tablet_replica.cc:442] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:13:23.810523 30407 catalog_manager.cc:5495] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c832b029a22043f1897594644bf1dbd7 (127.28.152.129). New cstate: current_term: 1 leader_uuid: "c832b029a22043f1897594644bf1dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } health_report { overall_health: UNKNOWN } } }
I20260926 02:13:23.811978 30957 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:13:23.812203 30957 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:13:23.828068 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.822533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:40269 (local address 127.0.0.1:55080)
0926 02:13:23.822743 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.822764 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.822906 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.823442 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.823448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.823459 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:13:23.823642 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.823650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.824684 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.824690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.825752 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.825764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.826489 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.826494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.826653 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.826709 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:13:23.827526 (+ 817us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:13:23.827557 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.827634 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.827785 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 02:13:23.828121 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.822889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.28.152.129:40269)
0926 02:13:23.823063 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.823068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.823085 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:13:23.823170 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.823174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.823268 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.823391 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:23.823773 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.824560 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.825904 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.826350 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.826458 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.826892 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:23.827422 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:23.827566 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.827674 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.827756 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:13:23.832449 30847 raft_consensus.cc:1261] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 1 FOLLOWER]: Refusing update from remote peer c832b029a22043f1897594644bf1dbd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:23.832719 30708 raft_consensus.cc:1261] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b [term 1 FOLLOWER]: Refusing update from remote peer c832b029a22043f1897594644bf1dbd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:23.833283 30918 consensus_queue.cc:1035] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:23.833779 30955 consensus_queue.cc:1035] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:23.837803 30919 tablet_replica.cc:439] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8: TxnStatusTablet state changed. Reason: New leader c832b029a22043f1897594644bf1dbd7. Latest consensus state: current_term: 1 leader_uuid: "c832b029a22043f1897594644bf1dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } } }
I20260926 02:13:23.838073 30919 tablet_replica.cc:442] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:23.838742 30955 tablet_replica.cc:439] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c832b029a22043f1897594644bf1dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } } }
I20260926 02:13:23.839962 30927 mvcc.cc:205] Tried to move back new op lower bound from 7333432540486062080 to 7333432540392189952. Current Snapshot: MvccSnapshot[applied={T|T < 7333432540486062080}]
I20260926 02:13:23.841822 30955 tablet_replica.cc:442] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:13:23.842247 30919 tablet_replica.cc:439] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c832b029a22043f1897594644bf1dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } } }
I20260926 02:13:23.842531 30919 tablet_replica.cc:442] T ba2ebfb8af9448f88dc367be6c6ba56f P 286bd6ee6c7a4aaeb09d731f34156bd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:23.853928 30932 tablet_replica.cc:439] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b: TxnStatusTablet state changed. Reason: New leader c832b029a22043f1897594644bf1dbd7. Latest consensus state: current_term: 1 leader_uuid: "c832b029a22043f1897594644bf1dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } } }
I20260926 02:13:23.854254 30932 tablet_replica.cc:442] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:23.854627 30955 tablet_replica.cc:439] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c832b029a22043f1897594644bf1dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } } }
I20260926 02:13:23.854902 30955 tablet_replica.cc:442] T ba2ebfb8af9448f88dc367be6c6ba56f P c832b029a22043f1897594644bf1dbd7: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:13:23.857185 30925 mvcc.cc:205] Tried to move back new op lower bound from 7333432540486062080 to 7333432540392189952. Current Snapshot: MvccSnapshot[applied={T|T < 7333432540486062080 or (T in {7333432540486062080})}]
I20260926 02:13:23.858243 30932 tablet_replica.cc:439] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c832b029a22043f1897594644bf1dbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 } } peers { permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 } } peers { permanent_uuid: "286bd6ee6c7a4aaeb09d731f34156bd8" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 42667 } } }
I20260926 02:13:23.858495 30932 tablet_replica.cc:442] T ba2ebfb8af9448f88dc367be6c6ba56f P d339244f27434bb7ba9377e5606a887b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:23.860337 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.848275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.28.152.131:42667)
0926 02:13:23.848821 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.848828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.848843 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:13:23.848952 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.848957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.849070 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.849253 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:23.849823 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.850921 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.852922 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.853619 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.853739 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.858366 (+ 4627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:23.859083 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:23.859229 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.859936 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.860049 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414}
I20260926 02:13:23.881402 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.874361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:56819 (local address 127.28.152.129:40269)
0926 02:13:23.874550 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.874557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.874747 (+ 190us) server_negotiation.cc:407] Connection header received
0926 02:13:23.875047 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.875053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.875144 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.875262 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:23.875869 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.876970 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.878993 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.879793 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.879939 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.880509 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:23.880665 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:23.880758 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.880910 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.881001 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:13:23.881465 30964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.873768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:40269 (local address 127.28.152.131:56819)
0926 02:13:23.874625 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.874647 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.874872 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.875285 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.875290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.875303 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:13:23.875673 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.875687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.877120 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.877167 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.878810 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.878831 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.880006 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.880014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.880205 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.880291 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:13:23.880775 (+ 484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:13:23.880787 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.880872 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.881037 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":784,"threads_started":1}
I20260926 02:13:23.896565 30564 raft_consensus.cc:1261] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7 [term 1 FOLLOWER]: Refusing update from remote peer 286bd6ee6c7a4aaeb09d731f34156bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:23.897572 30942 consensus_queue.cc:1035] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c832b029a22043f1897594644bf1dbd7" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:23.901017 30708 raft_consensus.cc:1261] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b [term 1 FOLLOWER]: Refusing update from remote peer 286bd6ee6c7a4aaeb09d731f34156bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:23.901731 30942 consensus_queue.cc:1035] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d339244f27434bb7ba9377e5606a887b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 35325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:23.962970 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.953802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:37107 (local address 127.28.152.131:42667)
0926 02:13:23.953941 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:23.953945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:23.955828 (+ 1883us) server_negotiation.cc:407] Connection header received
0926 02:13:23.956177 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:23.956182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:23.956276 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:23.956379 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:23.957320 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.958382 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.960552 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.961326 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.961472 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.962135 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:23.962275 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:23.962352 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:23.962586 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 02:13:23.962674 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 02:13:23.964210 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:23.954479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:42667 (local address 127.28.152.129:37107)
0926 02:13:23.955286 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:23.955307 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:23.955967 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:23.956565 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:23.956570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:23.956584 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:13:23.957093 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.957110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.958583 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.958588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.960371 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:23.960386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:23.961636 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:23.961643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:23.961837 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:23.961925 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:13:23.962447 (+ 522us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:13:23.962458 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:13:23.963636 (+ 1178us) client_negotiation.cc:240] Negotiation successful
0926 02:13:23.963795 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":419,"threads_started":1}
I20260926 02:13:24.026983 30407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:24.027259 30407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:24.031167 30407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 609c0e5c8fe642579c485e7684626764: Sending DeleteTablet for 3 replicas of tablet 5fb9329fd2124e509778834727ab45c0
I20260926 02:13:24.032780 30683 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb9329fd2124e509778834727ab45c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:24 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:40512
I20260926 02:13:24.033082 30935 tablet_replica.cc:330] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b: stopping tablet replica
I20260926 02:13:24.033311 30935 raft_consensus.cc:2229] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:24.033548 30935 raft_consensus.cc:2258] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:24.034037 30822 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb9329fd2124e509778834727ab45c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:24 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:48602
I20260926 02:13:24.034318 30937 tablet_replica.cc:330] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8: stopping tablet replica
I20260926 02:13:24.034942 30937 raft_consensus.cc:2229] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:13:24.035158 30935 ts_tablet_manager.cc:1905] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:24.035472 30937 raft_consensus.cc:2258] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:24.038355 30937 ts_tablet_manager.cc:1905] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:24.047549 30544 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb9329fd2124e509778834727ab45c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:24 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:55074
I20260926 02:13:24.047901 30936 tablet_replica.cc:330] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7: stopping tablet replica
I20260926 02:13:24.048125 30936 raft_consensus.cc:2229] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:24.048346 30936 raft_consensus.cc:2258] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:24.049844 30936 ts_tablet_manager.cc:1905] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:24.065886 30937 ts_tablet_manager.cc:1918] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:13:24.066087 30937 log.cc:1196] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/5fb9329fd2124e509778834727ab45c0
I20260926 02:13:24.066601 30937 ts_tablet_manager.cc:1939] T 5fb9329fd2124e509778834727ab45c0 P 286bd6ee6c7a4aaeb09d731f34156bd8: Deleting consensus metadata
I20260926 02:13:24.067973 30391 catalog_manager.cc:4841] TS 286bd6ee6c7a4aaeb09d731f34156bd8 (127.28.152.131:42667): tablet 5fb9329fd2124e509778834727ab45c0 (table test-table [id=fb58ac51952043c0bc6749e3d3dcc9c3]) successfully deleted
I20260926 02:13:24.072155 30935 ts_tablet_manager.cc:1918] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:13:24.072345 30935 log.cc:1196] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/5fb9329fd2124e509778834727ab45c0
I20260926 02:13:24.072820 30935 ts_tablet_manager.cc:1939] T 5fb9329fd2124e509778834727ab45c0 P d339244f27434bb7ba9377e5606a887b: Deleting consensus metadata
I20260926 02:13:24.073740 30936 ts_tablet_manager.cc:1918] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:13:24.073930 30936 log.cc:1196] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/5fb9329fd2124e509778834727ab45c0
I20260926 02:13:24.074429 30936 ts_tablet_manager.cc:1939] T 5fb9329fd2124e509778834727ab45c0 P c832b029a22043f1897594644bf1dbd7: Deleting consensus metadata
I20260926 02:13:24.075662 30391 catalog_manager.cc:4841] TS c832b029a22043f1897594644bf1dbd7 (127.28.152.129:40269): tablet 5fb9329fd2124e509778834727ab45c0 (table test-table [id=fb58ac51952043c0bc6749e3d3dcc9c3]) successfully deleted
I20260926 02:13:24.079344 30391 catalog_manager.cc:4841] TS d339244f27434bb7ba9377e5606a887b (127.28.152.130:35325): tablet 5fb9329fd2124e509778834727ab45c0 (table test-table [id=fb58ac51952043c0bc6749e3d3dcc9c3]) successfully deleted
Sep 26 02:13:24 dist-test-slave-hllj krb5kdc[30362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:13:24.105840 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:24.080735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.28.152.129:40269)
0926 02:13:24.080941 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:24.080948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:24.084752 (+ 3804us) server_negotiation.cc:407] Connection header received
0926 02:13:24.085367 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:24.085373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:24.085473 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:24.085554 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:24.087758 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:24.088872 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:24.091101 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:24.091763 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:24.091862 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:24.097897 (+ 6035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:24.097918 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:24.097940 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:24.097995 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:24.102078 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:24.103041 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:24.103052 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:24.103056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:24.103216 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:24.103683 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:24.103689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:24.103692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:24.104246 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:24.104567 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:24.105186 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 02:13:24.105444 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 02:13:24.107362 30609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55088
I20260926 02:13:24.145901 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:24.121599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.28.152.190:39735)
0926 02:13:24.121841 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:24.121848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:24.128939 (+ 7091us) server_negotiation.cc:407] Connection header received
0926 02:13:24.129519 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:24.129525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:24.129614 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:24.129688 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:24.131575 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:24.132670 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:24.134611 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:24.135245 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:24.135342 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:24.138809 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:24.138815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:24.138821 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:24.138866 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:24.142500 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:24.143372 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:24.143378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:24.143382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:24.143543 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:24.143947 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:24.143952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:24.143956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:24.144295 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:24.144538 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:24.145271 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 02:13:24.145540 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 02:13:24.147509 30407 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37586
I20260926 02:13:24.151069 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 30478
I20260926 02:13:24.192610 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:24.154011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.28.152.129:40269)
0926 02:13:24.154189 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:24.154195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:24.157589 (+ 3394us) server_negotiation.cc:407] Connection header received
0926 02:13:24.157804 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:24.157810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:24.157901 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:24.158008 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:24.177891 (+ 19883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:24.178803 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:24.180800 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:24.181749 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:24.181907 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:24.185160 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:24.185167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:24.185172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:24.185209 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:24.189291 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:24.190177 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:24.190184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:24.190190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:24.190350 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:24.190792 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:24.190798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:24.190801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:24.191192 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:24.191420 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:24.191974 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 02:13:24.192211 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 02:13:24.193917 30609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55092
W20260926 02:13:24.194502 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 30478: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:13:24.194672 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 30478
I20260926 02:13:24.212145 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 30618
Sep 26 02:13:24 dist-test-slave-hllj krb5kdc[30362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:13:24.228152 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:24.213208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.28.152.130:35325)
0926 02:13:24.213390 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:24.213396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:24.213869 (+ 473us) server_negotiation.cc:407] Connection header received
0926 02:13:24.214051 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:24.214057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:24.214135 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:24.214216 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:24.215423 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:24.216164 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:24.217585 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:24.218029 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:24.218139 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:24.222258 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:24.222279 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:24.222299 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:24.222337 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:24.225192 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:24.225971 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:24.225979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:24.225982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:24.226139 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:24.226532 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:24.226537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:24.226539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:24.226888 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:24.227217 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:24.227501 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 02:13:24.227725 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 02:13:24.228878 30748 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40530
W20260926 02:13:24.229633 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 30618: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:13:24.229816 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 30618
I20260926 02:13:24.248437 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 30757
Sep 26 02:13:24 dist-test-slave-hllj krb5kdc[30362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:13:24.264689 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:24.249191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.28.152.131:42667)
0926 02:13:24.249396 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:24.249402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:24.249514 (+ 112us) server_negotiation.cc:407] Connection header received
0926 02:13:24.249725 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:24.249728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:24.249795 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:24.249893 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:24.251106 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:24.251858 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:24.253156 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:24.253597 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:24.253711 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:24.258215 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:24.258231 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:24.258252 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:24.258297 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:24.261653 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:24.262562 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:24.262570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:24.262572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:24.262726 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:24.263119 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:24.263122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:24.263124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:24.263515 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:24.263753 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:24.264149 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 02:13:24.264347 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 02:13:24.265461 30887 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48628
W20260926 02:13:24.266084 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 30757: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:13:24.266219 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 30757
I20260926 02:13:24.281610 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 30375
I20260926 02:13:24.282802 30447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:24.395531 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 30375
2026-09-26T02:13:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4518 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:13:24 dist-test-slave-hllj krb5kdc[30981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:13:24 dist-test-slave-hllj krb5kdc[30981](info): set up 2 sockets
Sep 26 02:13:24 dist-test-slave-hllj krb5kdc[30981](info): commencing operation
krb5kdc: starting...
W20260926 02:13:25.398679 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:13:25 dist-test-slave-hllj krb5kdc[30981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:13:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:13:25.584616 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:33973
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:33973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:25.912143 30994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:25.912514 30994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:25.912654 30994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:25.920104 30994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:25.920255 30994 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:13:25.920315 30994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:25.920413 30994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:25.920481 30994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:25.920567 30994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:13:25.930984 30994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:33973
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:33973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:25.932399 30994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:25.933892 30994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:25.946455 30999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:25.946483 31002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:25.946439 31000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:25.947273 30994 server_base.cc:1034] running on GCE node
I20260926 02:13:25.948104 30994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:25.949747 30994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:25.951007 30994 hybrid_clock.cc:648] HybridClock initialized: now 1790388805950922 us; error 52 us; skew 500 ppm
I20260926 02:13:25.951397 30994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:25 dist-test-slave-hllj krb5kdc[30981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388805, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:25.954766 30994 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:13:25.956462 30994 webserver.cc:460] Webserver started at http://127.28.152.190:42417/ using document root <none> and password file <none>
I20260926 02:13:25.957276 30994 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:25.957407 30994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:25.957726 30994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:25.960987 30994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "a320e48175b24d8390e0617f4280d1a0"
format_stamp: "Formatted at 2026-09-26 02:13:25 on dist-test-slave-hllj"
server_key: "1cfa95ce2cb053a4b61eef555ee93583"
server_key_iv: "a6736d3a12394e46711e3acbbc788422"
server_key_version: "encryptionkey@0"
I20260926 02:13:25.961898 30994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "a320e48175b24d8390e0617f4280d1a0"
format_stamp: "Formatted at 2026-09-26 02:13:25 on dist-test-slave-hllj"
server_key: "1cfa95ce2cb053a4b61eef555ee93583"
server_key_iv: "a6736d3a12394e46711e3acbbc788422"
server_key_version: "encryptionkey@0"
I20260926 02:13:25.967624 30994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:13:25.971401 31009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:25.973037 30994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:13:25.973260 30994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "a320e48175b24d8390e0617f4280d1a0"
format_stamp: "Formatted at 2026-09-26 02:13:25 on dist-test-slave-hllj"
server_key: "1cfa95ce2cb053a4b61eef555ee93583"
server_key_iv: "a6736d3a12394e46711e3acbbc788422"
server_key_version: "encryptionkey@0"
I20260926 02:13:25.973476 30994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:25.996381 30994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:26.000820 30994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:26.001084 30994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:26.021827 30994 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:33973
I20260926 02:13:26.021845 31070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:33973 every 8 connection(s)
I20260926 02:13:26.023414 30994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:26.027442 31071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:26.033160 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 30994
I20260926 02:13:26.033366 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:26.034211 29282 external_mini_cluster.cc:1428] Setting key 36d0bfe4069a798e9c34c57f74c31fa9
I20260926 02:13:26.039734 31071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0: Bootstrap starting.
I20260926 02:13:26.042990 31071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:26.044087 31071 log.cc:824] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:26.047233 31071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0: No bootstrap required, opened a new log
Sep 26 02:13:26 dist-test-slave-hllj krb5kdc[30981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:26.055506 31071 raft_consensus.cc:348] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a320e48175b24d8390e0617f4280d1a0" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33973 } }
I20260926 02:13:26.055922 31071 raft_consensus.cc:374] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:26.056032 31071 raft_consensus.cc:729] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a320e48175b24d8390e0617f4280d1a0, State: Initialized, Role: FOLLOWER
I20260926 02:13:26.056768 31071 consensus_queue.cc:260] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a320e48175b24d8390e0617f4280d1a0" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33973 } }
I20260926 02:13:26.057077 31071 raft_consensus.cc:388] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:26.057233 31071 raft_consensus.cc:482] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:26.057421 31071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:26.058338 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:26.036582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.28.152.190:33973)
0926 02:13:26.037525 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:26.037539 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:26.037601 (+ 62us) server_negotiation.cc:407] Connection header received
0926 02:13:26.039382 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:26.039408 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:26.039890 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:26.040309 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:26.041874 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:26.043252 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:26.044589 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:26.045402 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:26.045588 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:26.050149 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:26.050158 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:26.050182 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:26.050231 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:26.053585 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:26.054387 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:26.054399 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:26.054411 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:26.054602 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:26.055000 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:26.055004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:26.055007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:26.055591 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:26.055865 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:26.056356 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 02:13:26.056849 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":440,"threads_started":1}
I20260926 02:13:26.060081 31071 raft_consensus.cc:504] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a320e48175b24d8390e0617f4280d1a0" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33973 } }
I20260926 02:13:26.060587 31071 leader_election.cc:302] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a320e48175b24d8390e0617f4280d1a0; no voters:
I20260926 02:13:26.061215 31071 leader_election.cc:288] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:26.061291 31076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:26.062026 31076 raft_consensus.cc:686] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [term 1 LEADER]: Becoming Leader. State: Replica: a320e48175b24d8390e0617f4280d1a0, State: Running, Role: LEADER
I20260926 02:13:26.062656 31076 consensus_queue.cc:237] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a320e48175b24d8390e0617f4280d1a0" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33973 } }
I20260926 02:13:26.063584 31071 sys_catalog.cc:564] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:26.065496 31078 sys_catalog.cc:455] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a320e48175b24d8390e0617f4280d1a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a320e48175b24d8390e0617f4280d1a0" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33973 } } }
I20260926 02:13:26.065897 31078 sys_catalog.cc:458] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:26.066444 31077 sys_catalog.cc:455] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a320e48175b24d8390e0617f4280d1a0. Latest consensus state: current_term: 1 leader_uuid: "a320e48175b24d8390e0617f4280d1a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a320e48175b24d8390e0617f4280d1a0" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33973 } } }
I20260926 02:13:26.066771 31077 sys_catalog.cc:458] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:26.067962 31085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:26.072997 31085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:26.082825 31085 catalog_manager.cc:1347] Generated new cluster ID: 370cc2524a9b4425b85057c4642c41a5
I20260926 02:13:26.082955 31085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:26.094928 31085 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:26.096858 31085 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:26.107759 31085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0: Generated new TSK 0
I20260926 02:13:26.108945 31085 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:13:26.157171 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33973
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:26.503355 31097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:26.503654 31097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:26.503778 31097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:26.511111 31097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:26.511232 31097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:26.511339 31097 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:13:26.511413 31097 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:13:26.511521 31097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:13:26.521490 31097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.152.190:33973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:26.522861 31097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:26.524185 31097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:26.537436 31103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:26.537577 31102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:26.537703 31105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:26.538662 31097 server_base.cc:1034] running on GCE node
I20260926 02:13:26.539305 31097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:26.540279 31097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:26.541477 31097 hybrid_clock.cc:648] HybridClock initialized: now 1790388806541407 us; error 60 us; skew 500 ppm
I20260926 02:13:26.541863 31097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:26 dist-test-slave-hllj krb5kdc[30981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388806, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:26.544895 31097 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:13:26.546427 31097 webserver.cc:460] Webserver started at http://127.28.152.129:34565/ using document root <none> and password file <none>
I20260926 02:13:26.547116 31097 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:26.547225 31097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:26.547533 31097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:26.550477 31097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035"
format_stamp: "Formatted at 2026-09-26 02:13:26 on dist-test-slave-hllj"
server_key: "ee077c94ba2a5c2877edc9e52a1ab5dc"
server_key_iv: "0760e78261316a91475a0d85db6e4ec9"
server_key_version: "encryptionkey@0"
I20260926 02:13:26.551304 31097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035"
format_stamp: "Formatted at 2026-09-26 02:13:26 on dist-test-slave-hllj"
server_key: "ee077c94ba2a5c2877edc9e52a1ab5dc"
server_key_iv: "0760e78261316a91475a0d85db6e4ec9"
server_key_version: "encryptionkey@0"
I20260926 02:13:26.556483 31097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260926 02:13:26.560177 31112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:26.561625 31097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:13:26.561774 31097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035"
format_stamp: "Formatted at 2026-09-26 02:13:26 on dist-test-slave-hllj"
server_key: "ee077c94ba2a5c2877edc9e52a1ab5dc"
server_key_iv: "0760e78261316a91475a0d85db6e4ec9"
server_key_version: "encryptionkey@0"
I20260926 02:13:26.561961 31097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:26.578053 31097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:26.582165 31097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:26.582425 31097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:26.585070 31097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:26.585207 31097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:13:26.585317 31097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:26.585404 31097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:13:26 dist-test-slave-hllj krb5kdc[30981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388806, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:26.608953 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:26.585710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:42231 (local address 127.28.152.190:33973)
0926 02:13:26.588203 (+ 2493us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:26.588210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:26.588233 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:13:26.589807 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:26.589813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:26.589902 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:26.590002 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:26.592340 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:26.593249 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:26.596326 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:26.596819 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:26.596929 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:26.602379 (+ 5450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:26.602386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:26.602390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:26.602428 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:26.604835 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:26.605983 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:26.605989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:26.605992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:26.606133 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:26.606795 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:26.606800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:26.606803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:26.607171 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:26.607328 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:26.608463 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0926 02:13:26.608684 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2367,"thread_start_us":2265,"threads_started":1}
I20260926 02:13:26.610704 31122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:26.586430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33973 (local address 127.28.152.129:42231)
0926 02:13:26.587334 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:26.587391 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:26.589506 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:26.590311 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:26.590325 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:26.591132 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:26.592137 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:26.592164 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:26.593438 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:26.593445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:26.596126 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:26.596147 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:26.597078 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:26.597091 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:26.597468 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:26.598335 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:26.598372 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:26.602056 (+ 3684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:26.605011 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:26.605024 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:26.605049 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:26.605823 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:26.606262 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:26.606268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:26.606272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:26.606608 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:26.607333 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:26.607346 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:26.607933 (+ 587us) client_negotiation.cc:769] Sending connection context
0926 02:13:26.608332 (+ 399us) client_negotiation.cc:240] Negotiation successful
0926 02:13:26.608991 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":347,"threads_started":1}
I20260926 02:13:26.631683 31197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:26.617715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33973 (local address 127.28.152.129:50181)
0926 02:13:26.618360 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:26.618379 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:26.618530 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:26.618916 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:26.618921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:26.619533 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:26.619928 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:26.619943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:26.620962 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:26.620968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:26.622786 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:26.622806 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:26.623595 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:26.623601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:26.623793 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:26.624430 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:26.624452 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:26.625674 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:26.628846 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:26.628851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:26.628854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:26.629428 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:26.629840 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:26.629845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:26.629849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:26.629977 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:26.630613 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:26.630617 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:26.630812 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 02:13:26.630942 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 02:13:26.631279 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":334,"threads_started":1}
I20260926 02:13:26.632437 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:26.617684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:50181 (local address 127.28.152.190:33973)
0926 02:13:26.617839 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:26.617843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:26.618470 (+ 627us) server_negotiation.cc:407] Connection header received
0926 02:13:26.618664 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:26.618668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:26.618753 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:26.618854 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:26.620100 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:26.620837 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:26.622948 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:26.623399 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:26.623505 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:26.625935 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:26.625942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:26.625947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:26.626008 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:26.628718 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:26.629564 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:26.629569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:26.629573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:26.629735 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:26.630087 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:26.630091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:26.630093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:26.630405 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:26.630555 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:26.632045 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0926 02:13:26.632228 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:13:26.641431 31097 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:45125
I20260926 02:13:26.641467 31232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:45125 every 8 connection(s)
I20260926 02:13:26.642694 31097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:13:26.644735 31233 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33973
I20260926 02:13:26.644963 31233 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:26.645720 31233 heartbeater.cc:511] Master 127.28.152.190:33973 requested a full tablet report, sending...
I20260926 02:13:26.647816 31026 ts_manager.cc:194] Registered new tserver with Master: 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125)
I20260926 02:13:26.647936 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 31097
I20260926 02:13:26.648113 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:13:26.648703 29282 external_mini_cluster.cc:1428] Setting key c42d56be900076025dc7e3cf00309ff6
I20260926 02:13:26.649562 31026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:42231
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:13:26.689811 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33973
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:27.016983 31237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:27.017306 31237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:27.017441 31237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:27.026901 31237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:27.027053 31237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:27.027180 31237 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:13:27.027247 31237 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:13:27.027345 31237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:13:27.043040 31237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.152.190:33973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:27.044632 31237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:27.046051 31237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:27.059631 31242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:27.059789 31245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:27.060179 31243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:27.060472 31237 server_base.cc:1034] running on GCE node
I20260926 02:13:27.061084 31237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:27.062156 31237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:27.063390 31237 hybrid_clock.cc:648] HybridClock initialized: now 1790388807063324 us; error 49 us; skew 500 ppm
I20260926 02:13:27.063740 31237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:27 dist-test-slave-hllj krb5kdc[30981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:27.067073 31237 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:13:27.068578 31237 webserver.cc:460] Webserver started at http://127.28.152.130:44339/ using document root <none> and password file <none>
I20260926 02:13:27.069316 31237 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:27.069450 31237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:27.069761 31237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:27.072759 31237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "a2cedc3998784c2b96e430d920ae7692"
format_stamp: "Formatted at 2026-09-26 02:13:27 on dist-test-slave-hllj"
server_key: "f9058db373022ccac68b0374db00c218"
server_key_iv: "a663f09609f42a3575ca2a6bfa7c824d"
server_key_version: "encryptionkey@0"
I20260926 02:13:27.073570 31237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "a2cedc3998784c2b96e430d920ae7692"
format_stamp: "Formatted at 2026-09-26 02:13:27 on dist-test-slave-hllj"
server_key: "f9058db373022ccac68b0374db00c218"
server_key_iv: "a663f09609f42a3575ca2a6bfa7c824d"
server_key_version: "encryptionkey@0"
I20260926 02:13:27.079314 31237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:13:27.083223 31252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:27.084661 31237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:13:27.084824 31237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "a2cedc3998784c2b96e430d920ae7692"
format_stamp: "Formatted at 2026-09-26 02:13:27 on dist-test-slave-hllj"
server_key: "f9058db373022ccac68b0374db00c218"
server_key_iv: "a663f09609f42a3575ca2a6bfa7c824d"
server_key_version: "encryptionkey@0"
I20260926 02:13:27.085017 31237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:27.103175 31237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:27.107846 31237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:27.108155 31237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:27.111078 31237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:27.111202 31237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:27.111330 31237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:27.111405 31237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:13:27 dist-test-slave-hllj krb5kdc[30981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:27.138101 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.111832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:39289 (local address 127.28.152.190:33973)
0926 02:13:27.112052 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:27.112059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:27.113863 (+ 1804us) server_negotiation.cc:407] Connection header received
0926 02:13:27.116829 (+ 2966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:27.116837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:27.116939 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:27.117033 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:27.120236 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.121198 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.124246 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.124817 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.124969 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.130278 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:27.130287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:27.130293 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:27.130346 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:27.133960 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:27.135198 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:27.135202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:27.135205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:27.135383 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:27.136014 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:27.136019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:27.136023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:27.136320 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:27.136468 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:27.137636 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0926 02:13:27.137830 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:13:27.139858 31261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.112631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33973 (local address 127.28.152.130:39289)
0926 02:13:27.113631 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:27.113692 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:27.116400 (+ 2708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:27.117734 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:27.117751 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:27.118762 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:27.119940 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.119983 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.121416 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.121423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.124033 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.124055 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.125326 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.125335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.125656 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.126464 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:27.126500 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:27.129962 (+ 3462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:27.134170 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:27.134179 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:27.134197 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:27.135027 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:27.135513 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:27.135520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:27.135524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:27.135877 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:27.136478 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:27.136489 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:27.137086 (+ 597us) client_negotiation.cc:769] Sending connection context
0926 02:13:27.137504 (+ 418us) client_negotiation.cc:240] Negotiation successful
0926 02:13:27.138157 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":350,"threads_started":1}
I20260926 02:13:27.157756 31237 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:36897
I20260926 02:13:27.157801 31371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:36897 every 8 connection(s)
I20260926 02:13:27.159143 31237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:13:27.161293 31372 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33973
I20260926 02:13:27.161536 31372 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:27.162441 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.147402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:33987 (local address 127.28.152.190:33973)
0926 02:13:27.147642 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:27.147649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:27.148130 (+ 481us) server_negotiation.cc:407] Connection header received
0926 02:13:27.148374 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:27.148379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:27.148469 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:27.148569 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:27.149772 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.150930 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.152930 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.153640 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.153782 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.156470 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:27.156477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:27.156482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:27.156529 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:27.159551 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:27.160511 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:27.160517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:27.160521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:27.160662 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:27.161042 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:27.161047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:27.161050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:27.161470 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:27.161603 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:27.162056 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 02:13:27.162225 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:13:27.162407 31372 heartbeater.cc:511] Master 127.28.152.190:33973 requested a full tablet report, sending...
I20260926 02:13:27.162806 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.147416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33973 (local address 127.28.152.130:33987)
0926 02:13:27.148019 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:27.148038 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:27.148210 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:27.148627 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:27.148632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:27.149252 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:27.149587 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.149612 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.151082 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.151089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.152749 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.152769 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.154091 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.154101 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.154314 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.155066 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:27.155101 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:27.156244 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:27.159725 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:27.159734 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:27.159738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:27.160354 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:27.160750 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:27.160755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:27.160758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:27.160919 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:27.161579 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:27.161583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:27.161807 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 02:13:27.161980 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 02:13:27.162335 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":323,"threads_started":1}
I20260926 02:13:27.164310 31025 ts_manager.cc:194] Registered new tserver with Master: a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897)
I20260926 02:13:27.165205 31025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:39289
I20260926 02:13:27.168006 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 31237
I20260926 02:13:27.168164 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:13:27.168578 29282 external_mini_cluster.cc:1428] Setting key d32fa799592806e0eca1295ef12ae832
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:13:27.208813 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33973
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:27.532052 31376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:27.532346 31376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:27.532445 31376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:27.539796 31376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:27.539916 31376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:27.539999 31376 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:13:27.540048 31376 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:13:27.540115 31376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:13:27.550094 31376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.152.190:33973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:27.551362 31376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:27.552700 31376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:27.566272 31381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:27.566299 31384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:27.566829 31382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:27.567234 31376 server_base.cc:1034] running on GCE node
I20260926 02:13:27.567922 31376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:27.568953 31376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:27.570175 31376 hybrid_clock.cc:648] HybridClock initialized: now 1790388807570099 us; error 71 us; skew 500 ppm
I20260926 02:13:27.570511 31376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:27 dist-test-slave-hllj krb5kdc[30981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:27.573637 31376 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:13:27.575181 31376 webserver.cc:460] Webserver started at http://127.28.152.131:40207/ using document root <none> and password file <none>
I20260926 02:13:27.575866 31376 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:27.575968 31376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:27.576247 31376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:27.579512 31376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "7ad0967fae494e02a12bb06f9d69ec6e"
format_stamp: "Formatted at 2026-09-26 02:13:27 on dist-test-slave-hllj"
server_key: "195cafb8e9a94b8d304aa89a8cbf6bce"
server_key_iv: "2bfcf5f83440761ece59952f2830192e"
server_key_version: "encryptionkey@0"
I20260926 02:13:27.580324 31376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "7ad0967fae494e02a12bb06f9d69ec6e"
format_stamp: "Formatted at 2026-09-26 02:13:27 on dist-test-slave-hllj"
server_key: "195cafb8e9a94b8d304aa89a8cbf6bce"
server_key_iv: "2bfcf5f83440761ece59952f2830192e"
server_key_version: "encryptionkey@0"
I20260926 02:13:27.585914 31376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 02:13:27.589681 31391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:27.591289 31376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:13:27.591468 31376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "7ad0967fae494e02a12bb06f9d69ec6e"
format_stamp: "Formatted at 2026-09-26 02:13:27 on dist-test-slave-hllj"
server_key: "195cafb8e9a94b8d304aa89a8cbf6bce"
server_key_iv: "2bfcf5f83440761ece59952f2830192e"
server_key_version: "encryptionkey@0"
I20260926 02:13:27.591666 31376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:27.610965 31376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:27.615434 31376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:27.615715 31376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:27.618542 31376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:27.618652 31376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:27.618769 31376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:27.618841 31376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:13:27 dist-test-slave-hllj krb5kdc[30981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:27.644251 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.619334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:58319 (local address 127.28.152.190:33973)
0926 02:13:27.619543 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:27.619551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:27.620996 (+ 1445us) server_negotiation.cc:407] Connection header received
0926 02:13:27.623558 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:27.623564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:27.623662 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:27.623767 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:27.626608 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.627576 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.630595 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.631105 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.631250 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.637102 (+ 5852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:27.637109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:27.637114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:27.637194 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:27.640482 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:27.641836 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:27.641840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:27.641843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:27.641987 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:27.642454 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:27.642460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:27.642464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:27.642766 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:27.642904 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:27.643666 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 02:13:27.643898 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:13:27.645537 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.619902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33973 (local address 127.28.152.131:58319)
0926 02:13:27.620788 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:27.620844 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:27.623106 (+ 2262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:27.624139 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:27.624153 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:27.625177 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:27.626341 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.626378 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.627752 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.627757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.630403 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.630420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.631548 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.631557 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.631842 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.632671 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:27.632708 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:27.636794 (+ 4086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:27.640991 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:27.641003 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:27.641025 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:27.641669 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:27.642112 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:27.642117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:27.642119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:27.642327 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:27.642899 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:27.642907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:27.643268 (+ 361us) client_negotiation.cc:769] Sending connection context
0926 02:13:27.643509 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 02:13:27.644040 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":343,"threads_started":1}
I20260926 02:13:27.653642 31233 heartbeater.cc:503] Master 127.28.152.190:33973 was elected leader, sending a full tablet report...
I20260926 02:13:27.667814 31376 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:41993
I20260926 02:13:27.667863 31510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:41993 every 8 connection(s)
I20260926 02:13:27.669207 31376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:13:27.671538 31511 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33973
I20260926 02:13:27.671756 31511 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:27.672569 31511 heartbeater.cc:511] Master 127.28.152.190:33973 requested a full tablet report, sending...
I20260926 02:13:27.674082 31025 ts_manager.cc:194] Registered new tserver with Master: 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993)
I20260926 02:13:27.674834 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.659970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:50031 (local address 127.28.152.190:33973)
0926 02:13:27.660170 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:27.660177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:27.660923 (+ 746us) server_negotiation.cc:407] Connection header received
0926 02:13:27.661120 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:27.661158 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:27.661236 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:27.661316 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:27.662777 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.663764 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.665995 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.666502 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.666617 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.668945 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:27.668950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:27.668953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:27.668986 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:27.672000 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:27.672796 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:27.672800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:27.672803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:27.672935 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:27.673322 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:27.673325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:27.673327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:27.673722 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:27.673871 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:27.674247 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 02:13:27.674476 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:13:27.674875 31488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.660035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33973 (local address 127.28.152.131:50031)
0926 02:13:27.660798 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:27.660817 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:27.660990 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:27.661521 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:27.661526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:27.662141 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:27.662564 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.662583 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.663956 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.663960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.665730 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.665751 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.666694 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.666701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.667290 (+ 589us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.667941 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:27.667966 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:27.668745 (+ 779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:27.672123 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:27.672128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:27.672132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:27.672684 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:27.673044 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:27.673049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:27.673053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:27.673232 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:27.673831 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:27.673836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:27.674025 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 02:13:27.674143 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 02:13:27.674461 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":472,"threads_started":1}
I20260926 02:13:27.674952 31025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:58319
I20260926 02:13:27.676826 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 31376
I20260926 02:13:27.676990 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:13:27.677682 29282 external_mini_cluster.cc:1428] Setting key 33768592c38361a71a6082b0a69541e4
I20260926 02:13:27.680393 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:13:27 dist-test-slave-hllj krb5kdc[30981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:13:27 dist-test-slave-hllj krb5kdc[30981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:27.725211 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.711028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.28.152.190:33973)
0926 02:13:27.711243 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:27.711250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:27.711403 (+ 153us) server_negotiation.cc:407] Connection header received
0926 02:13:27.711620 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:27.711625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:27.711685 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:27.711769 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:27.713267 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.714068 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.715299 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.715708 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.715815 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.719625 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:27.719631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:27.719635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:27.719672 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:27.722302 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:27.723117 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:27.723122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:27.723125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:27.723260 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:27.723655 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:27.723659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:27.723661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:27.723995 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:27.724229 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:27.724558 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 02:13:27.724791 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:13:27.729512 31025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:13:27.736249 31025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:13:27.759119 31525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.749092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.28.152.130:36897)
0926 02:13:27.750062 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:27.750072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:27.750769 (+ 697us) server_negotiation.cc:407] Connection header received
0926 02:13:27.750882 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:27.750890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:27.751178 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:27.751397 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:27.752277 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.753808 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.756492 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.757887 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.758065 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.758249 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:27.758590 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 02:13:27.758806 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":1605,"threads_started":1}
I20260926 02:13:27.759173 31528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.748742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36897 (local address 127.0.0.1:57042)
0926 02:13:27.750514 (+ 1772us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:27.750530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:27.750651 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:27.751640 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:27.751647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:27.751675 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:27.752069 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.752081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.753965 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.753970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.756303 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.756317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.758110 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.758117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.758320 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.758377 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 02:13:27.758524 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 02:13:27.758734 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1651,"mutex_wait_us":750,"thread_start_us":1339,"threads_started":1}
I20260926 02:13:27.759617 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.748742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:41993 (local address 127.0.0.1:38640)
0926 02:13:27.749568 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:27.749620 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:27.749803 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:27.752958 (+ 3155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:27.752962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:27.753039 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:27.753751 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.753765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.755088 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.755094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.757011 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.757022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.758654 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.758660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.758809 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.758840 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:13:27.758927 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 02:13:27.759061 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":27,"thread_start_us":1342,"threads_started":1}
I20260926 02:13:27.760878 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.749358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.28.152.131:41993)
0926 02:13:27.752378 (+ 3020us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:27.752387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:27.752435 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:13:27.752533 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:27.752540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:27.752766 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:27.753583 (+ 817us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:27.753945 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.754957 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.757180 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.758175 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.758356 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.758520 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:27.759258 (+ 738us) server_negotiation.cc:299] Negotiation successful
0926 02:13:27.759433 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2856,"thread_start_us":2154,"threads_started":1}
I20260926 02:13:27.761674 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.749881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45125 (local address 127.0.0.1:49990)
0926 02:13:27.751108 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:27.751126 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:27.751243 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:27.752938 (+ 1695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:27.752942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:27.752968 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:27.753777 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.753797 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.755646 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.755652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.758082 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:27.758099 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.760159 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.760165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:27.760340 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.760371 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:13:27.760507 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 02:13:27.760637 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":72,"thread_start_us":643,"threads_started":1}
I20260926 02:13:27.763026 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:27.750400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.28.152.129:45125)
0926 02:13:27.752094 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:27.752104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:27.752133 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:13:27.752225 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:27.752233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:27.752575 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:27.752807 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:27.753977 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.755477 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.758259 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:27.759937 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:27.760096 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:27.760258 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:27.762576 (+ 2318us) server_negotiation.cc:299] Negotiation successful
0926 02:13:27.762756 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1540,"thread_start_us":1420,"threads_started":1}
I20260926 02:13:27.763053 31302 tablet_service.cc:1467] Processing CreateTablet for tablet 01cd5eb6cf0d4319ac08267000a84852 (DEFAULT_TABLE table=test-table [id=1997aa62716a4beeb07f1bcd9bfac603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:27.764468 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01cd5eb6cf0d4319ac08267000a84852. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:27.765033 31441 tablet_service.cc:1467] Processing CreateTablet for tablet 01cd5eb6cf0d4319ac08267000a84852 (DEFAULT_TABLE table=test-table [id=1997aa62716a4beeb07f1bcd9bfac603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:27.766425 31441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01cd5eb6cf0d4319ac08267000a84852. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:27.766755 31163 tablet_service.cc:1467] Processing CreateTablet for tablet 01cd5eb6cf0d4319ac08267000a84852 (DEFAULT_TABLE table=test-table [id=1997aa62716a4beeb07f1bcd9bfac603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:27.768172 31163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01cd5eb6cf0d4319ac08267000a84852. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:27.778512 31530 tablet_bootstrap.cc:492] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: Bootstrap starting.
I20260926 02:13:27.781901 31530 tablet_bootstrap.cc:654] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:27.781862 31531 tablet_bootstrap.cc:492] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: Bootstrap starting.
I20260926 02:13:27.783457 31532 tablet_bootstrap.cc:492] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Bootstrap starting.
I20260926 02:13:27.784224 31530 log.cc:824] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:27.786033 31531 tablet_bootstrap.cc:654] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:27.786700 31532 tablet_bootstrap.cc:654] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:27.787317 31531 log.cc:824] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:27.787668 31530 tablet_bootstrap.cc:492] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: No bootstrap required, opened a new log
I20260926 02:13:27.787681 31532 log.cc:824] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:27.787982 31530 ts_tablet_manager.cc:1397] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:13:27.790668 31532 tablet_bootstrap.cc:492] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: No bootstrap required, opened a new log
I20260926 02:13:27.790928 31532 ts_tablet_manager.cc:1397] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 02:13:27.791778 31531 tablet_bootstrap.cc:492] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: No bootstrap required, opened a new log
I20260926 02:13:27.792131 31531 ts_tablet_manager.cc:1397] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 02:13:27.798646 31532 raft_consensus.cc:348] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } }
I20260926 02:13:27.798612 31530 raft_consensus.cc:348] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } }
I20260926 02:13:27.799048 31532 raft_consensus.cc:374] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:27.799163 31532 raft_consensus.cc:729] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1dd5d7bb1a4b4fb14c6481c995a035, State: Initialized, Role: FOLLOWER
I20260926 02:13:27.799185 31530 raft_consensus.cc:374] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:27.799276 31530 raft_consensus.cc:729] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad0967fae494e02a12bb06f9d69ec6e, State: Initialized, Role: FOLLOWER
I20260926 02:13:27.799786 31532 consensus_queue.cc:260] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } }
I20260926 02:13:27.799952 31530 consensus_queue.cc:260] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } }
I20260926 02:13:27.802224 31532 ts_tablet_manager.cc:1428] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 02:13:27.803020 31531 raft_consensus.cc:348] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } }
I20260926 02:13:27.804111 31531 raft_consensus.cc:374] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:27.804227 31531 raft_consensus.cc:729] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cedc3998784c2b96e430d920ae7692, State: Initialized, Role: FOLLOWER
I20260926 02:13:27.804947 31531 consensus_queue.cc:260] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } }
I20260926 02:13:27.806159 31372 heartbeater.cc:503] Master 127.28.152.190:33973 was elected leader, sending a full tablet report...
I20260926 02:13:27.807126 31511 heartbeater.cc:503] Master 127.28.152.190:33973 was elected leader, sending a full tablet report...
I20260926 02:13:27.808187 31531 ts_tablet_manager.cc:1428] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260926 02:13:27.808192 31530 ts_tablet_manager.cc:1428] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: Time spent starting tablet: real 0.020s user 0.007s sys 0.011s
W20260926 02:13:27.895013 31234 tablet.cc:2369] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:13:27.911226 31373 tablet.cc:2369] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:13:27.920862 31512 tablet.cc:2369] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:13:28.030668 31538 raft_consensus.cc:482] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:28.030987 31538 raft_consensus.cc:504] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } }
I20260926 02:13:28.032687 31538 leader_election.cc:288] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993), 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125)
I20260926 02:13:28.041443 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.033448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:42271 (local address 127.28.152.129:45125)
0926 02:13:28.033668 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:28.033675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:28.034950 (+ 1275us) server_negotiation.cc:407] Connection header received
0926 02:13:28.035085 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:28.035089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:28.035146 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:28.035224 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:28.035833 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.036828 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.039253 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.040447 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.040643 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.040705 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:28.041045 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 02:13:28.041207 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:13:28.041443 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.033828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:37145 (local address 127.28.152.131:41993)
0926 02:13:28.034014 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:28.034020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:28.034663 (+ 643us) server_negotiation.cc:407] Connection header received
0926 02:13:28.034851 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:28.034855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:28.034914 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:28.034999 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:28.035610 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.036653 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.039269 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.040447 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.040612 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.040684 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:28.041053 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 02:13:28.041198 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:13:28.041640 31539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.032929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:41993 (local address 127.28.152.130:37145)
0926 02:13:28.034516 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:28.034548 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:28.034719 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:28.035044 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:28.035049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:28.035074 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:28.035452 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.035468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.036779 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.036783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.039099 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.039108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.040670 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.040679 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.040864 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.040918 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 02:13:28.041008 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 02:13:28.041174 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1458,"mutex_wait_us":60,"thread_start_us":1258,"threads_started":1}
I20260926 02:13:28.041913 31540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.032948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45125 (local address 127.28.152.130:42271)
0926 02:13:28.034854 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:28.034866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:28.034965 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:28.035265 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:28.035270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:28.035289 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:28.035696 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.035705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.036949 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.036954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.039081 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.039097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.040687 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.040692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.040865 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.040919 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 02:13:28.041008 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 02:13:28.041171 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1784,"mutex_wait_us":47,"thread_start_us":1609,"threads_started":1}
I20260926 02:13:28.042654 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cd5eb6cf0d4319ac08267000a84852" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" is_pre_election: true
I20260926 02:13:28.042806 31184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cd5eb6cf0d4319ac08267000a84852" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" is_pre_election: true
I20260926 02:13:28.043181 31463 raft_consensus.cc:2454] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cedc3998784c2b96e430d920ae7692 in term 0.
I20260926 02:13:28.043298 31184 raft_consensus.cc:2454] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cedc3998784c2b96e430d920ae7692 in term 0.
I20260926 02:13:28.043888 31254 leader_election.cc:302] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ad0967fae494e02a12bb06f9d69ec6e, a2cedc3998784c2b96e430d920ae7692; no voters:
I20260926 02:13:28.044325 31538 raft_consensus.cc:2781] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:28.044445 31538 raft_consensus.cc:482] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:28.044517 31538 raft_consensus.cc:3037] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:28.046883 31538 raft_consensus.cc:504] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } }
I20260926 02:13:28.047597 31538 leader_election.cc:288] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 election: Requested vote from peers 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993), 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125)
I20260926 02:13:28.048015 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cd5eb6cf0d4319ac08267000a84852" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad0967fae494e02a12bb06f9d69ec6e"
I20260926 02:13:28.048049 31184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01cd5eb6cf0d4319ac08267000a84852" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035"
I20260926 02:13:28.048203 31463 raft_consensus.cc:3037] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:28.048244 31184 raft_consensus.cc:3037] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:28.050462 31463 raft_consensus.cc:2454] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cedc3998784c2b96e430d920ae7692 in term 1.
I20260926 02:13:28.050540 31184 raft_consensus.cc:2454] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cedc3998784c2b96e430d920ae7692 in term 1.
I20260926 02:13:28.050959 31256 leader_election.cc:302] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1dd5d7bb1a4b4fb14c6481c995a035, a2cedc3998784c2b96e430d920ae7692; no voters:
I20260926 02:13:28.051250 31538 raft_consensus.cc:2781] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:28.051723 31538 raft_consensus.cc:686] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 1 LEADER]: Becoming Leader. State: Replica: a2cedc3998784c2b96e430d920ae7692, State: Running, Role: LEADER
I20260926 02:13:28.052230 31538 consensus_queue.cc:237] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } }
I20260926 02:13:28.056708 31025 catalog_manager.cc:5495] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2cedc3998784c2b96e430d920ae7692 (127.28.152.130). New cstate: current_term: 1 leader_uuid: "a2cedc3998784c2b96e430d920ae7692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } health_report { overall_health: UNKNOWN } } }
Sep 26 02:13:28 dist-test-slave-hllj krb5kdc[30981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388805, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:28.156616 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.143639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.28.152.190:33973)
0926 02:13:28.143809 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:28.143814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:28.144317 (+ 503us) server_negotiation.cc:407] Connection header received
0926 02:13:28.144469 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:28.144474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:28.144544 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:28.144621 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:28.145814 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.146561 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.147810 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.148234 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.148341 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.151637 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:28.151642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:28.151646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:28.151675 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:28.154039 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:28.154713 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:28.154718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:28.154720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:28.154839 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:28.155169 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:28.155174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:28.155177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:28.155446 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:28.155576 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:28.155990 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 02:13:28.156305 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":62}
I20260926 02:13:28.156651 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.143667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33973 (local address 127.0.0.1:53250)
0926 02:13:28.144210 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:28.144226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:28.144364 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:28.144662 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:28.144666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:28.145410 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:28.145676 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.145686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.146680 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.146684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.147677 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.147690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.148416 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.148422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.148574 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.149218 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:28.149235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:28.151403 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:28.154176 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:28.154192 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:28.154201 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:28.154597 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:28.154939 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:28.154944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:28.154948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:28.155075 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:28.155577 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:28.155586 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:28.155749 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 02:13:28.155889 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 02:13:28.156165 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":291,"threads_started":1}
I20260926 02:13:28.162335 31025 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:53250:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:13:28.163548 31025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:13:28.171252 31302 tablet_service.cc:1467] Processing CreateTablet for tablet 1d91b6712fae47d8b6ff6d5ee6c51019 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff11be1b94ad4d4593658835aef91b41]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:13:28.171069 31163 tablet_service.cc:1467] Processing CreateTablet for tablet 1d91b6712fae47d8b6ff6d5ee6c51019 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff11be1b94ad4d4593658835aef91b41]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:13:28.171478 31441 tablet_service.cc:1467] Processing CreateTablet for tablet 1d91b6712fae47d8b6ff6d5ee6c51019 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ff11be1b94ad4d4593658835aef91b41]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:13:28.172021 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d91b6712fae47d8b6ff6d5ee6c51019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:28.172072 31163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d91b6712fae47d8b6ff6d5ee6c51019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:28.172183 31441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d91b6712fae47d8b6ff6d5ee6c51019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:28.177834 31531 tablet_bootstrap.cc:492] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: Bootstrap starting.
I20260926 02:13:28.179045 31532 tablet_bootstrap.cc:492] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Bootstrap starting.
I20260926 02:13:28.179440 31531 tablet_bootstrap.cc:654] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:28.180361 31532 tablet_bootstrap.cc:654] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:28.181420 31531 tablet_bootstrap.cc:492] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: No bootstrap required, opened a new log
I20260926 02:13:28.181660 31531 ts_tablet_manager.cc:1397] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:13:28.182323 31531 raft_consensus.cc:348] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.182502 31531 raft_consensus.cc:374] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:28.182561 31532 tablet_bootstrap.cc:492] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035: No bootstrap required, opened a new log
I20260926 02:13:28.182603 31531 raft_consensus.cc:729] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cedc3998784c2b96e430d920ae7692, State: Initialized, Role: FOLLOWER
I20260926 02:13:28.182700 31532 ts_tablet_manager.cc:1397] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:13:28.182801 31531 consensus_queue.cc:260] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.183091 31530 tablet_bootstrap.cc:492] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e: Bootstrap starting.
I20260926 02:13:28.183166 31538 tablet_replica.cc:439] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } }
I20260926 02:13:28.183418 31538 tablet_replica.cc:442] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:28.183446 31532 raft_consensus.cc:348] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.183733 31532 raft_consensus.cc:374] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:28.183866 31532 raft_consensus.cc:729] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1dd5d7bb1a4b4fb14c6481c995a035, State: Initialized, Role: FOLLOWER
I20260926 02:13:28.184206 31532 consensus_queue.cc:260] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.184628 31530 tablet_bootstrap.cc:654] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:28.184713 31536 tablet_replica.cc:439] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } }
I20260926 02:13:28.185082 31536 tablet_replica.cc:442] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:28.185693 31532 ts_tablet_manager.cc:1428] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 02:13:28.186225 31531 ts_tablet_manager.cc:1428] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260926 02:13:28.187862 31530 tablet_bootstrap.cc:492] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e: No bootstrap required, opened a new log
I20260926 02:13:28.188009 31530 ts_tablet_manager.cc:1397] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 02:13:28.188768 31530 raft_consensus.cc:348] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.188966 31530 raft_consensus.cc:374] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:28.189056 31530 raft_consensus.cc:729] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad0967fae494e02a12bb06f9d69ec6e, State: Initialized, Role: FOLLOWER
I20260926 02:13:28.189270 31530 consensus_queue.cc:260] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.189647 31537 tablet_replica.cc:439] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } }
I20260926 02:13:28.189798 31530 ts_tablet_manager.cc:1428] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 02:13:28.189927 31537 tablet_replica.cc:442] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:28.273610 31538 raft_consensus.cc:482] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:28.273828 31538 raft_consensus.cc:504] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.274432 31538 leader_election.cc:288] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125), 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993)
I20260926 02:13:28.275022 31184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d91b6712fae47d8b6ff6d5ee6c51019" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" is_pre_election: true
I20260926 02:13:28.275010 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d91b6712fae47d8b6ff6d5ee6c51019" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" is_pre_election: true
I20260926 02:13:28.275251 31184 raft_consensus.cc:2454] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cedc3998784c2b96e430d920ae7692 in term 0.
I20260926 02:13:28.275257 31463 raft_consensus.cc:2454] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cedc3998784c2b96e430d920ae7692 in term 0.
I20260926 02:13:28.275729 31254 leader_election.cc:302] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e1dd5d7bb1a4b4fb14c6481c995a035, 7ad0967fae494e02a12bb06f9d69ec6e, a2cedc3998784c2b96e430d920ae7692; no voters:
I20260926 02:13:28.276016 31538 raft_consensus.cc:2781] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:28.276126 31538 raft_consensus.cc:482] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:28.276228 31538 raft_consensus.cc:3037] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:28.278553 31538 raft_consensus.cc:504] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.279166 31538 leader_election.cc:288] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 election: Requested vote from peers 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125), 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993)
I20260926 02:13:28.279632 31184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d91b6712fae47d8b6ff6d5ee6c51019" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035"
I20260926 02:13:28.279726 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d91b6712fae47d8b6ff6d5ee6c51019" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad0967fae494e02a12bb06f9d69ec6e"
I20260926 02:13:28.279902 31184 raft_consensus.cc:3037] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:28.279934 31463 raft_consensus.cc:3037] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:28.282842 31463 raft_consensus.cc:2454] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cedc3998784c2b96e430d920ae7692 in term 1.
I20260926 02:13:28.282847 31184 raft_consensus.cc:2454] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cedc3998784c2b96e430d920ae7692 in term 1.
I20260926 02:13:28.283314 31256 leader_election.cc:302] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1dd5d7bb1a4b4fb14c6481c995a035, a2cedc3998784c2b96e430d920ae7692; no voters:
I20260926 02:13:28.283592 31538 raft_consensus.cc:2781] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:28.283794 31538 raft_consensus.cc:686] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [term 1 LEADER]: Becoming Leader. State: Replica: a2cedc3998784c2b96e430d920ae7692, State: Running, Role: LEADER
I20260926 02:13:28.284024 31538 consensus_queue.cc:237] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.285724 31541 tablet_replica.cc:439] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: TxnStatusTablet state changed. Reason: New leader a2cedc3998784c2b96e430d920ae7692. Latest consensus state: current_term: 1 leader_uuid: "a2cedc3998784c2b96e430d920ae7692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } }
I20260926 02:13:28.286015 31541 tablet_replica.cc:442] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:13:28.286646 31553 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:13:28.286891 31553 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:13:28.287122 31024 catalog_manager.cc:5495] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2cedc3998784c2b96e430d920ae7692 (127.28.152.130). New cstate: current_term: 1 leader_uuid: "a2cedc3998784c2b96e430d920ae7692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } health_report { overall_health: UNKNOWN } } }
I20260926 02:13:28.317842 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.310853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36897 (local address 127.0.0.1:57050)
0926 02:13:28.311105 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:28.311126 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:28.311301 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:28.312079 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:28.312083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:28.312121 (+ 38us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:13:28.312314 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.312323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.313940 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.313944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.314943 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.314956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.315973 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.315978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.316131 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.316195 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:13:28.317401 (+ 1206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:13:28.317422 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 02:13:28.317478 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 02:13:28.317620 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 02:13:28.318004 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.310936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.28.152.130:36897)
0926 02:13:28.311584 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:28.311590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:28.311611 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:13:28.311708 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:28.311714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:28.311827 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:28.312011 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:28.312467 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.313754 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.315097 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.315787 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.315922 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.316437 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:28.317296 (+ 859us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:28.317452 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:28.317517 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 02:13:28.317606 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":356,"threads_started":1}
I20260926 02:13:28.326721 31463 raft_consensus.cc:1261] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Refusing update from remote peer a2cedc3998784c2b96e430d920ae7692: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:28.327378 31184 raft_consensus.cc:1261] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Refusing update from remote peer a2cedc3998784c2b96e430d920ae7692: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:28.328109 31541 consensus_queue.cc:1035] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:28.328811 31538 consensus_queue.cc:1035] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:28.341092 31537 tablet_replica.cc:439] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e: TxnStatusTablet state changed. Reason: New leader a2cedc3998784c2b96e430d920ae7692. Latest consensus state: current_term: 1 leader_uuid: "a2cedc3998784c2b96e430d920ae7692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } }
I20260926 02:13:28.341427 31537 tablet_replica.cc:442] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:28.344480 31557 mvcc.cc:205] Tried to move back new op lower bound from 7333432558894854144 to 7333432558733475840. Current Snapshot: MvccSnapshot[applied={T|T < 7333432558894854144}]
I20260926 02:13:28.345319 31538 tablet_replica.cc:439] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2cedc3998784c2b96e430d920ae7692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } }
I20260926 02:13:28.345670 31538 tablet_replica.cc:442] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:13:28.352545 31536 tablet_replica.cc:439] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035: TxnStatusTablet state changed. Reason: New leader a2cedc3998784c2b96e430d920ae7692. Latest consensus state: current_term: 1 leader_uuid: "a2cedc3998784c2b96e430d920ae7692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } }
I20260926 02:13:28.352842 31536 tablet_replica.cc:442] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:28.355405 31541 tablet_replica.cc:439] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2cedc3998784c2b96e430d920ae7692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } }
I20260926 02:13:28.355749 31541 tablet_replica.cc:442] T 1d91b6712fae47d8b6ff6d5ee6c51019 P a2cedc3998784c2b96e430d920ae7692: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:13:28.358273 31537 tablet_replica.cc:439] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2cedc3998784c2b96e430d920ae7692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } }
I20260926 02:13:28.358500 31537 tablet_replica.cc:442] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 7ad0967fae494e02a12bb06f9d69ec6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:28.362347 31536 tablet_replica.cc:439] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2cedc3998784c2b96e430d920ae7692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } } }
I20260926 02:13:28.362641 31536 tablet_replica.cc:442] T 1d91b6712fae47d8b6ff6d5ee6c51019 P 2e1dd5d7bb1a4b4fb14c6481c995a035: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:13:28.371362 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.366135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.28.152.130:36897)
0926 02:13:28.366338 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:28.366344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:28.366423 (+ 79us) server_negotiation.cc:407] Connection header received
0926 02:13:28.366633 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:28.366640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:28.366731 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:28.366831 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:28.367349 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.368112 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.369626 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.370088 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.370200 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.370649 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:28.370755 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:28.370837 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:28.370985 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 02:13:28.371076 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 02:13:28.388617 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.382771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:36537 (local address 127.28.152.130:36897)
0926 02:13:28.382938 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:28.382942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:28.383597 (+ 655us) server_negotiation.cc:407] Connection header received
0926 02:13:28.383774 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:28.383777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:28.383856 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:28.383926 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:28.384480 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.385297 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.386781 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.387229 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.387340 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.387812 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:28.387916 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:28.387994 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:28.388118 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 02:13:28.388192 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":46}
I20260926 02:13:28.388653 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.382830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36897 (local address 127.28.152.130:36537)
0926 02:13:28.383495 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:28.383512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:28.383657 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:28.383949 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:28.383953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:28.383963 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:13:28.384349 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.384357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.385430 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.385434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.386641 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.386655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.387413 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.387420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.387545 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.387613 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:13:28.388017 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:13:28.388028 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:13:28.388095 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 02:13:28.388245 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":18,"thread_start_us":356,"threads_started":1}
I20260926 02:13:28.398283 31184 raft_consensus.cc:1261] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Refusing update from remote peer a2cedc3998784c2b96e430d920ae7692: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:28.398419 31463 raft_consensus.cc:1261] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Refusing update from remote peer a2cedc3998784c2b96e430d920ae7692: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:28.398921 31538 consensus_queue.cc:1035] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:28.399219 31565 consensus_queue.cc:1035] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:28.467209 31025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:28.467549 31025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:28.471024 31025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0: Sending DeleteTablet for 3 replicas of tablet 01cd5eb6cf0d4319ac08267000a84852
I20260926 02:13:28.472266 31441 tablet_service.cc:1514] Processing DeleteTablet for tablet 01cd5eb6cf0d4319ac08267000a84852 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:28 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:38640
I20260926 02:13:28.472268 31302 tablet_service.cc:1514] Processing DeleteTablet for tablet 01cd5eb6cf0d4319ac08267000a84852 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:28 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:57042
I20260926 02:13:28.473047 31163 tablet_service.cc:1514] Processing DeleteTablet for tablet 01cd5eb6cf0d4319ac08267000a84852 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:28 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:49990
I20260926 02:13:28.473275 31575 tablet_replica.cc:330] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: stopping tablet replica
I20260926 02:13:28.473701 31575 raft_consensus.cc:2229] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:28.474071 31575 raft_consensus.cc:2258] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:28.474809 31574 tablet_replica.cc:330] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: stopping tablet replica
I20260926 02:13:28.475309 31574 raft_consensus.cc:2229] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:13:28.476012 31575 ts_tablet_manager.cc:1905] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:28.475983 31574 raft_consensus.cc:2258] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:28.475966 31025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:13:28.476658 31025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:13:28.484033 31575 ts_tablet_manager.cc:1918] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:13:28.484282 31575 log.cc:1196] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/01cd5eb6cf0d4319ac08267000a84852
I20260926 02:13:28.484824 31575 ts_tablet_manager.cc:1939] T 01cd5eb6cf0d4319ac08267000a84852 P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting consensus metadata
I20260926 02:13:28.485625 31162 tablet_service.cc:1467] Processing CreateTablet for tablet 34d0d7c8e7b548d48879cdc20cd83884 (DEFAULT_TABLE table=test-table [id=8ef07bded2924c70ade63fbe17292249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:28.485883 31302 tablet_service.cc:1467] Processing CreateTablet for tablet 34d0d7c8e7b548d48879cdc20cd83884 (DEFAULT_TABLE table=test-table [id=8ef07bded2924c70ade63fbe17292249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:28.486232 31162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d0d7c8e7b548d48879cdc20cd83884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:28.486348 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d0d7c8e7b548d48879cdc20cd83884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:28.487425 31576 tablet_replica.cc:330] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: stopping tablet replica
I20260926 02:13:28.487902 31576 raft_consensus.cc:2229] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:28.488289 31576 raft_consensus.cc:2258] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:28.489176 31441 tablet_service.cc:1467] Processing CreateTablet for tablet 34d0d7c8e7b548d48879cdc20cd83884 (DEFAULT_TABLE table=test-table [id=8ef07bded2924c70ade63fbe17292249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:28.489611 31441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d0d7c8e7b548d48879cdc20cd83884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:28.493147 31532 tablet_bootstrap.cc:492] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Bootstrap starting.
I20260926 02:13:28.494522 31574 ts_tablet_manager.cc:1905] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:28.494818 31532 tablet_bootstrap.cc:654] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:28.495282 31531 tablet_bootstrap.cc:492] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692: Bootstrap starting.
I20260926 02:13:28.496987 31531 tablet_bootstrap.cc:654] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:28.498968 31530 tablet_bootstrap.cc:492] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e: Bootstrap starting.
I20260926 02:13:28.500845 31530 tablet_bootstrap.cc:654] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:28.501679 31011 catalog_manager.cc:4841] TS 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993): tablet 01cd5eb6cf0d4319ac08267000a84852 (table test-table [id=1997aa62716a4beeb07f1bcd9bfac603]) successfully deleted
I20260926 02:13:28.502636 31531 tablet_bootstrap.cc:492] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692: No bootstrap required, opened a new log
I20260926 02:13:28.502919 31531 ts_tablet_manager.cc:1397] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 02:13:28.503835 31531 raft_consensus.cc:348] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.504245 31531 raft_consensus.cc:374] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:28.504387 31531 raft_consensus.cc:729] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cedc3998784c2b96e430d920ae7692, State: Initialized, Role: FOLLOWER
I20260926 02:13:28.504743 31531 consensus_queue.cc:260] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.506400 31531 ts_tablet_manager.cc:1428] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 02:13:28.514860 31576 ts_tablet_manager.cc:1905] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:28.517113 31532 tablet_bootstrap.cc:492] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035: No bootstrap required, opened a new log
I20260926 02:13:28.517433 31532 ts_tablet_manager.cc:1397] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.003s
I20260926 02:13:28.517642 31530 tablet_bootstrap.cc:492] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e: No bootstrap required, opened a new log
I20260926 02:13:28.517730 31574 ts_tablet_manager.cc:1918] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:13:28.517861 31530 ts_tablet_manager.cc:1397] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.002s
I20260926 02:13:28.517983 31574 log.cc:1196] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/01cd5eb6cf0d4319ac08267000a84852
I20260926 02:13:28.518345 31532 raft_consensus.cc:348] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.518602 31574 ts_tablet_manager.cc:1939] T 01cd5eb6cf0d4319ac08267000a84852 P a2cedc3998784c2b96e430d920ae7692: Deleting consensus metadata
I20260926 02:13:28.518668 31532 raft_consensus.cc:374] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:28.518611 31530 raft_consensus.cc:348] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.518793 31532 raft_consensus.cc:729] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1dd5d7bb1a4b4fb14c6481c995a035, State: Initialized, Role: FOLLOWER
I20260926 02:13:28.518893 31530 raft_consensus.cc:374] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:28.519006 31530 raft_consensus.cc:729] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad0967fae494e02a12bb06f9d69ec6e, State: Initialized, Role: FOLLOWER
I20260926 02:13:28.519167 31532 consensus_queue.cc:260] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.519309 31530 consensus_queue.cc:260] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.519819 31532 ts_tablet_manager.cc:1428] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:13:28.520510 31530 ts_tablet_manager.cc:1428] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 02:13:28.520622 31012 catalog_manager.cc:4841] TS a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897): tablet 01cd5eb6cf0d4319ac08267000a84852 (table test-table [id=1997aa62716a4beeb07f1bcd9bfac603]) successfully deleted
I20260926 02:13:28.528208 31576 ts_tablet_manager.cc:1918] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:13:28.528374 31576 log.cc:1196] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/01cd5eb6cf0d4319ac08267000a84852
I20260926 02:13:28.528891 31576 ts_tablet_manager.cc:1939] T 01cd5eb6cf0d4319ac08267000a84852 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Deleting consensus metadata
I20260926 02:13:28.530836 31013 catalog_manager.cc:4841] TS 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125): tablet 01cd5eb6cf0d4319ac08267000a84852 (table test-table [id=1997aa62716a4beeb07f1bcd9bfac603]) successfully deleted
I20260926 02:13:28.561659 31537 raft_consensus.cc:482] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:28.561882 31537 raft_consensus.cc:504] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.563547 31537 leader_election.cc:288] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125), a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897)
I20260926 02:13:28.580905 31581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.563718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45125 (local address 127.28.152.131:53517)
0926 02:13:28.573671 (+ 9953us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:28.573699 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:28.573877 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:28.574435 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:28.574440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:28.574469 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:28.574768 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.574780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.576080 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.576085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.578208 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.578227 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.580020 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.580027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.580211 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.580246 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:13:28.580336 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 02:13:28.580480 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9825,"mutex_wait_us":36,"thread_start_us":10499,"threads_started":1}
I20260926 02:13:28.581310 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.564637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:49127 (local address 127.28.152.130:36897)
0926 02:13:28.564866 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:28.564873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:28.574680 (+ 9807us) server_negotiation.cc:407] Connection header received
0926 02:13:28.574815 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:28.574820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:28.574903 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:28.575013 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:28.575611 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.576722 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.578827 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.579925 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.580219 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.580294 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:28.580840 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 02:13:28.581012 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:13:28.581631 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.564352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:53517 (local address 127.28.152.129:45125)
0926 02:13:28.573955 (+ 9603us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:28.573961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:28.573980 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:13:28.574051 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:28.574055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:28.574147 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:28.574248 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:28.574937 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.575946 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.578372 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.579671 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.579853 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.579932 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:28.581194 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0926 02:13:28.581339 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9469,"thread_start_us":9594,"threads_started":1}
I20260926 02:13:28.582053 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.564070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36897 (local address 127.28.152.131:49127)
0926 02:13:28.574548 (+ 10478us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:28.574568 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:28.574686 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:28.575039 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:28.575043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:28.575069 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:28.575392 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.575404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.576860 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.576865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.578672 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.578681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.580450 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.580457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.580636 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.580667 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:13:28.581586 (+ 919us) client_negotiation.cc:240] Negotiation successful
0926 02:13:28.581691 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10345,"mutex_wait_us":1,"thread_start_us":9327,"threads_started":1}
I20260926 02:13:28.582557 31184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d0d7c8e7b548d48879cdc20cd83884" candidate_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" is_pre_election: true
I20260926 02:13:28.582809 31184 raft_consensus.cc:2454] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ad0967fae494e02a12bb06f9d69ec6e in term 0.
I20260926 02:13:28.582895 31322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d0d7c8e7b548d48879cdc20cd83884" candidate_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cedc3998784c2b96e430d920ae7692" is_pre_election: true
I20260926 02:13:28.583348 31322 raft_consensus.cc:2454] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ad0967fae494e02a12bb06f9d69ec6e in term 0.
I20260926 02:13:28.583943 31394 leader_election.cc:302] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ad0967fae494e02a12bb06f9d69ec6e, a2cedc3998784c2b96e430d920ae7692; no voters:
I20260926 02:13:28.584415 31537 raft_consensus.cc:2781] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:28.584606 31537 raft_consensus.cc:482] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:28.584702 31537 raft_consensus.cc:3037] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:28.586841 31537 raft_consensus.cc:504] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.587399 31537 leader_election.cc:288] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [CANDIDATE]: Term 1 election: Requested vote from peers 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125), a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897)
I20260926 02:13:28.587946 31322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d0d7c8e7b548d48879cdc20cd83884" candidate_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cedc3998784c2b96e430d920ae7692"
I20260926 02:13:28.587946 31184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d0d7c8e7b548d48879cdc20cd83884" candidate_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035"
I20260926 02:13:28.588155 31322 raft_consensus.cc:3037] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:28.588152 31184 raft_consensus.cc:3037] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:28.590559 31322 raft_consensus.cc:2454] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ad0967fae494e02a12bb06f9d69ec6e in term 1.
I20260926 02:13:28.590999 31394 leader_election.cc:302] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ad0967fae494e02a12bb06f9d69ec6e, a2cedc3998784c2b96e430d920ae7692; no voters:
I20260926 02:13:28.591058 31184 raft_consensus.cc:2454] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ad0967fae494e02a12bb06f9d69ec6e in term 1.
I20260926 02:13:28.591281 31537 raft_consensus.cc:2781] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:28.591760 31537 raft_consensus.cc:686] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 LEADER]: Becoming Leader. State: Replica: 7ad0967fae494e02a12bb06f9d69ec6e, State: Running, Role: LEADER
I20260926 02:13:28.592147 31537 consensus_queue.cc:237] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:28.595907 31024 catalog_manager.cc:5495] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131). New cstate: current_term: 1 leader_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } health_report { overall_health: HEALTHY } } }
I20260926 02:13:28.620476 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.613891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.28.152.131:41993)
0926 02:13:28.614448 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:28.614454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:28.614480 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:13:28.614584 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:28.614589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:28.614716 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:28.614908 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:28.615275 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.616423 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.617783 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.618409 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.618542 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.618975 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:28.619801 (+ 826us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:28.619966 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:28.620016 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 02:13:28.620106 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":295,"threads_started":1}
I20260926 02:13:28.635727 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.630284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:54065 (local address 127.28.152.130:36897)
0926 02:13:28.630430 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:28.630434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:28.630995 (+ 561us) server_negotiation.cc:407] Connection header received
0926 02:13:28.631179 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:28.631183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:28.631258 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:28.631338 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:28.631798 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.632579 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.633947 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.634466 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.634579 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.634969 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:28.635101 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:28.635184 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:28.635309 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 02:13:28.635398 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 02:13:28.635774 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.630317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36897 (local address 127.28.152.131:54065)
0926 02:13:28.630888 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:28.630902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:28.631049 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:28.631356 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:28.631360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:28.631371 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:13:28.631652 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.631661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.632704 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.632710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.633815 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.633826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.634638 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.634645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.634757 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.634812 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:13:28.635195 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:13:28.635207 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:13:28.635274 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 02:13:28.635409 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":313,"threads_started":1}
I20260926 02:13:28.647535 31322 raft_consensus.cc:1261] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Refusing update from remote peer 7ad0967fae494e02a12bb06f9d69ec6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:28.647768 31184 raft_consensus.cc:1261] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Refusing update from remote peer 7ad0967fae494e02a12bb06f9d69ec6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:28.648478 31583 consensus_queue.cc:1035] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:28.648839 31537 consensus_queue.cc:1035] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:28.652918 31558 mvcc.cc:205] Tried to move back new op lower bound from 7333432560213164032 to 7333432559996940288. Current Snapshot: MvccSnapshot[applied={T|T < 7333432560213164032}]
I20260926 02:13:28.653986 31559 mvcc.cc:205] Tried to move back new op lower bound from 7333432560213164032 to 7333432559996940288. Current Snapshot: MvccSnapshot[applied={T|T < 7333432560213164032}]
I20260926 02:13:28.696704 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.690475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:38935 (local address 127.28.152.131:41993)
0926 02:13:28.690636 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:28.690642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:28.690659 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:13:28.690723 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:28.690728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:28.690803 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:28.690882 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:28.691637 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.692607 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.694894 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.695490 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.695623 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.696037 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:28.696160 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:28.696231 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:28.696367 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 02:13:28.696444 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:13:28.696879 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:28.689761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:41993 (local address 127.28.152.130:38935)
0926 02:13:28.690153 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:28.690177 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:28.690279 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:28.691049 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:28.691055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:28.691068 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:13:28.691252 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.691264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.693086 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.693092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.694719 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:28.694735 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:28.695649 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:28.695656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:28.695838 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:28.695878 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:13:28.696256 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:13:28.696268 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:13:28.696341 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 02:13:28.696568 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
W20260926 02:13:28.698275 31558 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 02:13:30.309247 31024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:30.309545 31024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:30.313917 31024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0: Sending DeleteTablet for 3 replicas of tablet 34d0d7c8e7b548d48879cdc20cd83884
I20260926 02:13:30.314832 31162 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d0d7c8e7b548d48879cdc20cd83884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:30 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:49990
I20260926 02:13:30.314893 31441 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d0d7c8e7b548d48879cdc20cd83884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:30 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:38640
I20260926 02:13:30.314935 31301 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d0d7c8e7b548d48879cdc20cd83884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:30 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:57042
I20260926 02:13:30.326960 31615 tablet_replica.cc:330] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692: stopping tablet replica
I20260926 02:13:30.332242 31615 raft_consensus.cc:2229] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:30.332734 31615 raft_consensus.cc:2258] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:30.334106 31615 ts_tablet_manager.cc:1905] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:30.337313 31616 tablet_replica.cc:330] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035: stopping tablet replica
I20260926 02:13:30.337533 31616 raft_consensus.cc:2229] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:30.337900 31616 raft_consensus.cc:2258] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:30.338212 31614 tablet_replica.cc:330] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e: stopping tablet replica
I20260926 02:13:30.338480 31614 raft_consensus.cc:2229] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:13:30.339076 31614 raft_consensus.cc:2258] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:30.339313 31616 ts_tablet_manager.cc:1905] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:30.339994 31614 ts_tablet_manager.cc:1905] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:13:30 dist-test-slave-hllj krb5kdc[30981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388810, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260926 02:13:30.438220 29282 mini_kdc.cc:317] Time spent kinit for test-user: real 0.122s user 0.000s sys 0.014s
I20260926 02:13:30.438372 31616 ts_tablet_manager.cc:1918] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:13:30.438372 31614 ts_tablet_manager.cc:1918] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:13:30.438529 31616 log.cc:1196] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/34d0d7c8e7b548d48879cdc20cd83884
I20260926 02:13:30.438529 31614 log.cc:1196] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/34d0d7c8e7b548d48879cdc20cd83884
I20260926 02:13:30.438978 31616 ts_tablet_manager.cc:1939] T 34d0d7c8e7b548d48879cdc20cd83884 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Deleting consensus metadata
I20260926 02:13:30.438978 31614 ts_tablet_manager.cc:1939] T 34d0d7c8e7b548d48879cdc20cd83884 P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting consensus metadata
I20260926 02:13:30.439464 31615 ts_tablet_manager.cc:1918] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:13:30.439618 31615 log.cc:1196] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/34d0d7c8e7b548d48879cdc20cd83884
I20260926 02:13:30.440029 31615 ts_tablet_manager.cc:1939] T 34d0d7c8e7b548d48879cdc20cd83884 P a2cedc3998784c2b96e430d920ae7692: Deleting consensus metadata
I20260926 02:13:30.444247 31011 catalog_manager.cc:4841] TS 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993): tablet 34d0d7c8e7b548d48879cdc20cd83884 (table test-table [id=8ef07bded2924c70ade63fbe17292249]) successfully deleted
I20260926 02:13:30.444342 31013 catalog_manager.cc:4841] TS 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125): tablet 34d0d7c8e7b548d48879cdc20cd83884 (table test-table [id=8ef07bded2924c70ade63fbe17292249]) successfully deleted
Sep 26 02:13:30 dist-test-slave-hllj krb5kdc[30981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388810, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:30.462004 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:30.442667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.28.152.190:33973)
0926 02:13:30.443483 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:30.443488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:30.443514 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:13:30.443577 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:30.443582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:30.443660 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:30.443775 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:30.445346 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.446155 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.447591 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.448059 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.448171 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:30.456580 (+ 8409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:30.456586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:30.456590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:30.456637 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:30.459366 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:30.460163 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:30.460168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:30.460170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:30.460336 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:30.460733 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:30.460736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:30.460738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:30.461011 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:30.461274 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:30.461521 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 02:13:30.461697 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":541,"threads_started":1}
I20260926 02:13:30.463909 31012 catalog_manager.cc:4841] TS a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897): tablet 34d0d7c8e7b548d48879cdc20cd83884 (table test-table [id=8ef07bded2924c70ade63fbe17292249]) successfully deleted
I20260926 02:13:30.465710 31024 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:13:30.466269 31024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:13:30.472868 31162 tablet_service.cc:1467] Processing CreateTablet for tablet 6762c5939bb847859d3ef3c16d0233cb (DEFAULT_TABLE table=test-table [id=809e625cb5c440fea829f5ab3da9fb61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:30.472931 31301 tablet_service.cc:1467] Processing CreateTablet for tablet 6762c5939bb847859d3ef3c16d0233cb (DEFAULT_TABLE table=test-table [id=809e625cb5c440fea829f5ab3da9fb61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:30.473394 31301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6762c5939bb847859d3ef3c16d0233cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:30.473421 31441 tablet_service.cc:1467] Processing CreateTablet for tablet 6762c5939bb847859d3ef3c16d0233cb (DEFAULT_TABLE table=test-table [id=809e625cb5c440fea829f5ab3da9fb61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:30.473744 31441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6762c5939bb847859d3ef3c16d0233cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:30.477483 31162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6762c5939bb847859d3ef3c16d0233cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:30.478302 31625 tablet_bootstrap.cc:492] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692: Bootstrap starting.
I20260926 02:13:30.480348 31625 tablet_bootstrap.cc:654] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:30.481891 31626 tablet_bootstrap.cc:492] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e: Bootstrap starting.
I20260926 02:13:30.482537 31625 tablet_bootstrap.cc:492] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692: No bootstrap required, opened a new log
I20260926 02:13:30.482681 31625 ts_tablet_manager.cc:1397] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 02:13:30.483640 31626 tablet_bootstrap.cc:654] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:30.483778 31625 raft_consensus.cc:348] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.484104 31625 raft_consensus.cc:374] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:30.484238 31625 raft_consensus.cc:729] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cedc3998784c2b96e430d920ae7692, State: Initialized, Role: FOLLOWER
I20260926 02:13:30.484335 31628 tablet_bootstrap.cc:492] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035: Bootstrap starting.
I20260926 02:13:30.484661 31625 consensus_queue.cc:260] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.485255 31625 ts_tablet_manager.cc:1428] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:13:30.485984 31626 tablet_bootstrap.cc:492] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e: No bootstrap required, opened a new log
I20260926 02:13:30.486136 31626 ts_tablet_manager.cc:1397] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:13:30.486841 31628 tablet_bootstrap.cc:654] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:30.486841 31626 raft_consensus.cc:348] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.487082 31626 raft_consensus.cc:374] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:30.487214 31626 raft_consensus.cc:729] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad0967fae494e02a12bb06f9d69ec6e, State: Initialized, Role: FOLLOWER
I20260926 02:13:30.487478 31626 consensus_queue.cc:260] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.488051 31626 ts_tablet_manager.cc:1428] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:13:30.489495 31628 tablet_bootstrap.cc:492] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035: No bootstrap required, opened a new log
I20260926 02:13:30.489780 31628 ts_tablet_manager.cc:1397] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 02:13:30.490569 31628 raft_consensus.cc:348] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.490780 31628 raft_consensus.cc:374] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:30.490869 31628 raft_consensus.cc:729] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1dd5d7bb1a4b4fb14c6481c995a035, State: Initialized, Role: FOLLOWER
I20260926 02:13:30.491103 31628 consensus_queue.cc:260] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.491814 31628 ts_tablet_manager.cc:1428] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:13:30.567175 31631 raft_consensus.cc:482] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:30.567444 31631 raft_consensus.cc:504] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.569236 31631 leader_election.cc:288] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897), 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993)
I20260926 02:13:30.583444 31601 raft_consensus.cc:482] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:30.583663 31601 raft_consensus.cc:504] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.584353 31601 leader_election.cc:288] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125), 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993)
I20260926 02:13:30.584959 31462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6762c5939bb847859d3ef3c16d0233cb" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" is_pre_election: true
I20260926 02:13:30.585002 31184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6762c5939bb847859d3ef3c16d0233cb" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" is_pre_election: true
I20260926 02:13:30.585275 31462 raft_consensus.cc:2454] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cedc3998784c2b96e430d920ae7692 in term 0.
I20260926 02:13:30.585286 31184 raft_consensus.cc:2454] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cedc3998784c2b96e430d920ae7692 in term 0.
I20260926 02:13:30.585794 31254 leader_election.cc:302] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ad0967fae494e02a12bb06f9d69ec6e, a2cedc3998784c2b96e430d920ae7692; no voters:
I20260926 02:13:30.586117 31601 raft_consensus.cc:2781] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:30.586232 31601 raft_consensus.cc:482] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:30.586313 31601 raft_consensus.cc:3037] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:30.589233 31632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:30.569520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:53131 (local address 127.28.152.130:36897)
0926 02:13:30.582038 (+ 12518us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:30.582056 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:30.583091 (+ 1035us) server_negotiation.cc:407] Connection header received
0926 02:13:30.583289 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:30.583310 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:30.583385 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:30.583481 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:30.583960 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.585163 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.587207 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.588170 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.588356 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:30.588476 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:30.588734 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 02:13:30.588868 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12359,"thread_start_us":12568,"threads_started":1}
I20260926 02:13:30.589795 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:30.569491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36897 (local address 127.28.152.129:53131)
0926 02:13:30.582961 (+ 13470us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:30.582979 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:30.583131 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:30.583522 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:30.583527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:30.583552 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:30.583786 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:30.583794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.585326 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.585332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:30.587053 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:30.587065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.588371 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.588378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:30.588554 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:30.588588 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:13:30.588689 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 02:13:30.588827 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13327,"mutex_wait_us":1,"thread_start_us":13134,"threads_started":1}
I20260926 02:13:30.590693 31633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:30.569756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:41993 (local address 127.28.152.129:50193)
0926 02:13:30.583103 (+ 13347us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:30.583121 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:30.583278 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:30.584227 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:30.584232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:30.584257 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:30.584600 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:30.584613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.586109 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.586114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:30.588211 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:30.588225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.589960 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.589967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:30.590152 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:30.590183 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:13:30.590277 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 02:13:30.590416 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13206,"mutex_wait_us":50,"thread_start_us":12859,"threads_started":1}
I20260926 02:13:30.590530 31322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6762c5939bb847859d3ef3c16d0233cb" candidate_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cedc3998784c2b96e430d920ae7692" is_pre_election: true
I20260926 02:13:30.590777 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:30.569855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:50193 (local address 127.28.152.131:41993)
0926 02:13:30.581776 (+ 11921us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:30.581784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:30.583212 (+ 1428us) server_negotiation.cc:407] Connection header received
0926 02:13:30.583997 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:30.584002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:30.584089 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:30.584190 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:30.584791 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.585943 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.588408 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.589602 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.589764 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:30.589820 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:30.590324 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 02:13:30.590465 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11759,"thread_start_us":11610,"threads_started":1}
I20260926 02:13:30.591373 31462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6762c5939bb847859d3ef3c16d0233cb" candidate_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" is_pre_election: true
I20260926 02:13:30.591604 31462 raft_consensus.cc:2454] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e1dd5d7bb1a4b4fb14c6481c995a035 in term 0.
I20260926 02:13:30.592033 31114 leader_election.cc:302] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1dd5d7bb1a4b4fb14c6481c995a035, 7ad0967fae494e02a12bb06f9d69ec6e; no voters:
I20260926 02:13:30.592412 31631 raft_consensus.cc:2781] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:30.592511 31631 raft_consensus.cc:482] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:30.592582 31631 raft_consensus.cc:3037] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:30.638234 31631 raft_consensus.cc:504] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.638234 31601 raft_consensus.cc:504] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.638724 31322 raft_consensus.cc:2379] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2e1dd5d7bb1a4b4fb14c6481c995a035 in current term 1: Already voted for candidate a2cedc3998784c2b96e430d920ae7692 in this term.
I20260926 02:13:30.639190 31631 leader_election.cc:288] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [CANDIDATE]: Term 1 election: Requested vote from peers a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897), 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993)
I20260926 02:13:30.639838 31322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6762c5939bb847859d3ef3c16d0233cb" candidate_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cedc3998784c2b96e430d920ae7692"
I20260926 02:13:30.639901 31462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6762c5939bb847859d3ef3c16d0233cb" candidate_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad0967fae494e02a12bb06f9d69ec6e"
I20260926 02:13:30.640089 31462 raft_consensus.cc:3037] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:30.640126 31322 raft_consensus.cc:2379] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2e1dd5d7bb1a4b4fb14c6481c995a035 in current term 1: Already voted for candidate a2cedc3998784c2b96e430d920ae7692 in this term.
I20260926 02:13:30.640395 31601 leader_election.cc:288] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 election: Requested vote from peers 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125), 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993)
I20260926 02:13:30.641031 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6762c5939bb847859d3ef3c16d0233cb" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad0967fae494e02a12bb06f9d69ec6e"
I20260926 02:13:30.641151 31184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6762c5939bb847859d3ef3c16d0233cb" candidate_uuid: "a2cedc3998784c2b96e430d920ae7692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035"
I20260926 02:13:30.641424 31184 raft_consensus.cc:2379] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2cedc3998784c2b96e430d920ae7692 in current term 1: Already voted for candidate 2e1dd5d7bb1a4b4fb14c6481c995a035 in this term.
I20260926 02:13:30.642552 31256 leader_election.cc:302] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2cedc3998784c2b96e430d920ae7692; no voters: 2e1dd5d7bb1a4b4fb14c6481c995a035, 7ad0967fae494e02a12bb06f9d69ec6e
I20260926 02:13:30.642989 31601 raft_consensus.cc:2726] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 02:13:30.740980 31462 raft_consensus.cc:2454] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e1dd5d7bb1a4b4fb14c6481c995a035 in term 1.
I20260926 02:13:30.741706 31114 leader_election.cc:302] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e1dd5d7bb1a4b4fb14c6481c995a035, 7ad0967fae494e02a12bb06f9d69ec6e; no voters: a2cedc3998784c2b96e430d920ae7692
I20260926 02:13:30.742121 31631 raft_consensus.cc:2781] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:30.742734 31631 raft_consensus.cc:686] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 LEADER]: Becoming Leader. State: Replica: 2e1dd5d7bb1a4b4fb14c6481c995a035, State: Running, Role: LEADER
I20260926 02:13:30.743206 31631 consensus_queue.cc:237] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.747102 31025 catalog_manager.cc:5495] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129). New cstate: current_term: 1 leader_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } health_report { overall_health: UNKNOWN } } }
I20260926 02:13:30.796326 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:30.789846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.28.152.129:45125)
0926 02:13:30.790607 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:30.790612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:30.790643 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:13:30.790739 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:30.790742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:30.790820 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:30.790961 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:30.791393 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.792264 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.793768 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.794233 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.794364 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:30.794884 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:30.795563 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:30.795771 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:30.795894 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 02:13:30.795986 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":490,"threads_started":1}
I20260926 02:13:30.813437 31632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:30.807694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:35853 (local address 127.28.152.130:36897)
0926 02:13:30.807899 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:30.807907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:30.808344 (+ 437us) server_negotiation.cc:407] Connection header received
0926 02:13:30.808497 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:30.808500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:30.808616 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:30.808697 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:30.809666 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.810405 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.811777 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.812191 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.812320 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:30.812792 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:30.812923 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:30.812990 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:30.813082 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 02:13:30.813194 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:13:30.813544 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:30.807572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36897 (local address 127.28.152.129:35853)
0926 02:13:30.808235 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:30.808257 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:30.808388 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:30.808722 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:30.808725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:30.808734 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:13:30.809482 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:30.809495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.810560 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.810565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:30.811622 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:30.811634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.812395 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.812400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:30.812543 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:30.812612 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:13:30.812988 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:13:30.812995 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:13:30.813052 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 02:13:30.813238 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":338,"threads_started":1}
I20260926 02:13:30.822589 31322 raft_consensus.cc:1261] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Refusing update from remote peer 2e1dd5d7bb1a4b4fb14c6481c995a035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:30.823191 31462 raft_consensus.cc:1261] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Refusing update from remote peer 2e1dd5d7bb1a4b4fb14c6481c995a035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:30.823338 31636 consensus_queue.cc:1035] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:30.823856 31631 consensus_queue.cc:1035] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:30.854763 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:30.849072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:36715 (local address 127.28.152.129:45125)
0926 02:13:30.849265 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:30.849271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:30.849765 (+ 494us) server_negotiation.cc:407] Connection header received
0926 02:13:30.849953 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:30.849959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:30.850028 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:30.850096 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:30.850464 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.851213 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.852753 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.853241 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.853368 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:30.854212 (+ 844us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:30.854319 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:30.854388 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:30.854490 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 02:13:30.854554 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:13:30.854851 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:30.848939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45125 (local address 127.28.152.130:36715)
0926 02:13:30.849670 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:30.849690 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:30.849824 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:30.850109 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:30.850112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:30.850121 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:13:30.850363 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:30.850370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.851317 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.851320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:30.852580 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:30.852592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:30.853423 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:30.853430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:30.853575 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:30.853623 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:13:30.854405 (+ 782us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:13:30.854414 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 02:13:30.854473 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 02:13:30.854581 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":440,"threads_started":1}
I20260926 02:13:30.887859 31025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:30.888139 31025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:30.891863 31025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0: Sending DeleteTablet for 3 replicas of tablet 6762c5939bb847859d3ef3c16d0233cb
I20260926 02:13:30.892823 31162 tablet_service.cc:1514] Processing DeleteTablet for tablet 6762c5939bb847859d3ef3c16d0233cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:30 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:49990
I20260926 02:13:30.892951 31301 tablet_service.cc:1514] Processing DeleteTablet for tablet 6762c5939bb847859d3ef3c16d0233cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:30 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:57042
I20260926 02:13:30.893178 31616 tablet_replica.cc:330] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035: stopping tablet replica
I20260926 02:13:30.893388 31616 raft_consensus.cc:2229] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:13:30.893391 31441 tablet_service.cc:1514] Processing DeleteTablet for tablet 6762c5939bb847859d3ef3c16d0233cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:30 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:38640
I20260926 02:13:30.893669 31614 tablet_replica.cc:330] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e: stopping tablet replica
I20260926 02:13:30.893844 31614 raft_consensus.cc:2229] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:30.893890 31616 raft_consensus.cc:2258] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:30.893990 31614 raft_consensus.cc:2258] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:30.894210 31615 tablet_replica.cc:330] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692: stopping tablet replica
I20260926 02:13:30.894400 31615 raft_consensus.cc:2229] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:30.894578 31615 raft_consensus.cc:2258] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:30.895457 31025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 02:13:30.895919 31615 ts_tablet_manager.cc:1905] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 02:13:30.896070 31025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:13:30.898190 31616 ts_tablet_manager.cc:1905] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:30.900207 31614 ts_tablet_manager.cc:1905] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:30.903910 31162 tablet_service.cc:1467] Processing CreateTablet for tablet 32add0586466412bb6af559160f42c91 (DEFAULT_TABLE table=test-table [id=cfb5329ad2134b7e9e59211df1f2262d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:30.904085 31301 tablet_service.cc:1467] Processing CreateTablet for tablet 32add0586466412bb6af559160f42c91 (DEFAULT_TABLE table=test-table [id=cfb5329ad2134b7e9e59211df1f2262d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:30.905359 31441 tablet_service.cc:1467] Processing CreateTablet for tablet 32add0586466412bb6af559160f42c91 (DEFAULT_TABLE table=test-table [id=cfb5329ad2134b7e9e59211df1f2262d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:30.948567 31441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32add0586466412bb6af559160f42c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:30.948609 31162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32add0586466412bb6af559160f42c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:30.955998 31626 tablet_bootstrap.cc:492] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e: Bootstrap starting.
I20260926 02:13:30.957830 31301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32add0586466412bb6af559160f42c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:30.957835 31626 tablet_bootstrap.cc:654] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:30.961675 31628 tablet_bootstrap.cc:492] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Bootstrap starting.
I20260926 02:13:30.963449 31628 tablet_bootstrap.cc:654] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:30.966287 31628 tablet_bootstrap.cc:492] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035: No bootstrap required, opened a new log
I20260926 02:13:30.966351 31626 tablet_bootstrap.cc:492] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e: No bootstrap required, opened a new log
I20260926 02:13:30.966459 31628 ts_tablet_manager.cc:1397] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260926 02:13:30.966511 31626 ts_tablet_manager.cc:1397] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 02:13:30.967187 31628 raft_consensus.cc:348] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.967403 31628 raft_consensus.cc:374] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:30.967483 31628 raft_consensus.cc:729] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1dd5d7bb1a4b4fb14c6481c995a035, State: Initialized, Role: FOLLOWER
I20260926 02:13:30.967351 31626 raft_consensus.cc:348] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.967602 31626 raft_consensus.cc:374] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:30.967666 31626 raft_consensus.cc:729] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad0967fae494e02a12bb06f9d69ec6e, State: Initialized, Role: FOLLOWER
I20260926 02:13:30.967700 31628 consensus_queue.cc:260] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.968561 31628 ts_tablet_manager.cc:1428] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 02:13:30.968657 31626 consensus_queue.cc:260] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:30.971318 31626 ts_tablet_manager.cc:1428] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260926 02:13:31.010067 31647 raft_consensus.cc:482] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:31.010283 31647 raft_consensus.cc:504] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:31.011109 31647 leader_election.cc:288] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897), 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993)
I20260926 02:13:31.011838 31462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32add0586466412bb6af559160f42c91" candidate_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" is_pre_election: true
I20260926 02:13:31.012087 31462 raft_consensus.cc:2454] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e1dd5d7bb1a4b4fb14c6481c995a035 in term 0.
W20260926 02:13:31.012460 31115 leader_election.cc:341] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897): Not found: Tablet not found: 32add0586466412bb6af559160f42c91
I20260926 02:13:31.012739 31114 leader_election.cc:302] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e1dd5d7bb1a4b4fb14c6481c995a035, 7ad0967fae494e02a12bb06f9d69ec6e; no voters: a2cedc3998784c2b96e430d920ae7692
I20260926 02:13:31.013002 31647 raft_consensus.cc:2781] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:31.013170 31647 raft_consensus.cc:482] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:31.013266 31647 raft_consensus.cc:3037] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:31.048875 31616 ts_tablet_manager.cc:1918] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:13:31.048789 31647 raft_consensus.cc:504] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:31.049079 31616 log.cc:1196] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/6762c5939bb847859d3ef3c16d0233cb
I20260926 02:13:31.049602 31616 ts_tablet_manager.cc:1939] T 6762c5939bb847859d3ef3c16d0233cb P 2e1dd5d7bb1a4b4fb14c6481c995a035: Deleting consensus metadata
W20260926 02:13:31.050537 31115 leader_election.cc:341] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897): Not found: Tablet not found: 32add0586466412bb6af559160f42c91
I20260926 02:13:31.050915 31614 ts_tablet_manager.cc:1918] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:13:31.050968 31013 catalog_manager.cc:4841] TS 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125): tablet 6762c5939bb847859d3ef3c16d0233cb (table test-table [id=809e625cb5c440fea829f5ab3da9fb61]) successfully deleted
I20260926 02:13:31.051083 31614 log.cc:1196] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/6762c5939bb847859d3ef3c16d0233cb
I20260926 02:13:31.051406 31647 leader_election.cc:288] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [CANDIDATE]: Term 1 election: Requested vote from peers a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897), 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993)
I20260926 02:13:31.051501 31614 ts_tablet_manager.cc:1939] T 6762c5939bb847859d3ef3c16d0233cb P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting consensus metadata
I20260926 02:13:31.052441 31615 ts_tablet_manager.cc:1918] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:13:31.052582 31615 log.cc:1196] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/6762c5939bb847859d3ef3c16d0233cb
I20260926 02:13:31.053009 31615 ts_tablet_manager.cc:1939] T 6762c5939bb847859d3ef3c16d0233cb P a2cedc3998784c2b96e430d920ae7692: Deleting consensus metadata
I20260926 02:13:31.053206 31462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32add0586466412bb6af559160f42c91" candidate_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad0967fae494e02a12bb06f9d69ec6e"
I20260926 02:13:31.053419 31462 raft_consensus.cc:3037] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:31.056108 31654 tablet_bootstrap.cc:492] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692: Bootstrap starting.
I20260926 02:13:31.057542 31654 tablet_bootstrap.cc:654] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:31.060047 31654 tablet_bootstrap.cc:492] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692: No bootstrap required, opened a new log
I20260926 02:13:31.060213 31654 ts_tablet_manager.cc:1397] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 02:13:31.060988 31654 raft_consensus.cc:348] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:31.061241 31654 raft_consensus.cc:374] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:31.061337 31654 raft_consensus.cc:729] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cedc3998784c2b96e430d920ae7692, State: Initialized, Role: FOLLOWER
I20260926 02:13:31.061570 31654 consensus_queue.cc:260] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:31.062103 31654 ts_tablet_manager.cc:1428] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:13:31.089386 31011 catalog_manager.cc:4841] TS 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993): tablet 6762c5939bb847859d3ef3c16d0233cb (table test-table [id=809e625cb5c440fea829f5ab3da9fb61]) successfully deleted
I20260926 02:13:31.122187 31012 catalog_manager.cc:4841] TS a2cedc3998784c2b96e430d920ae7692 (127.28.152.130:36897): tablet 6762c5939bb847859d3ef3c16d0233cb (table test-table [id=809e625cb5c440fea829f5ab3da9fb61]) successfully deleted
I20260926 02:13:31.152093 31462 raft_consensus.cc:2454] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e1dd5d7bb1a4b4fb14c6481c995a035 in term 1.
I20260926 02:13:31.152858 31114 leader_election.cc:302] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e1dd5d7bb1a4b4fb14c6481c995a035, 7ad0967fae494e02a12bb06f9d69ec6e; no voters: a2cedc3998784c2b96e430d920ae7692
I20260926 02:13:31.153306 31647 raft_consensus.cc:2781] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:31.153491 31647 raft_consensus.cc:686] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 LEADER]: Becoming Leader. State: Replica: 2e1dd5d7bb1a4b4fb14c6481c995a035, State: Running, Role: LEADER
I20260926 02:13:31.153782 31647 consensus_queue.cc:237] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } }
I20260926 02:13:31.158133 31025 catalog_manager.cc:5495] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129). New cstate: current_term: 1 leader_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1dd5d7bb1a4b4fb14c6481c995a035" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 } health_report { overall_health: UNKNOWN } } }
I20260926 02:13:31.192291 31322 raft_consensus.cc:3037] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:31.192557 31462 raft_consensus.cc:1261] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Refusing update from remote peer 2e1dd5d7bb1a4b4fb14c6481c995a035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:31.193341 31636 consensus_queue.cc:1035] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad0967fae494e02a12bb06f9d69ec6e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:13:31.214006 31597 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 02:13:31.253096 31322 raft_consensus.cc:1261] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Refusing update from remote peer 2e1dd5d7bb1a4b4fb14c6481c995a035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:31.253837 31647 consensus_queue.cc:1035] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cedc3998784c2b96e430d920ae7692" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:32.817150 31025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:32.817377 31025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:32.822468 31025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a320e48175b24d8390e0617f4280d1a0: Sending DeleteTablet for 3 replicas of tablet 32add0586466412bb6af559160f42c91
I20260926 02:13:32.823436 31441 tablet_service.cc:1514] Processing DeleteTablet for tablet 32add0586466412bb6af559160f42c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:32 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:38640
I20260926 02:13:32.823503 31162 tablet_service.cc:1514] Processing DeleteTablet for tablet 32add0586466412bb6af559160f42c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:32 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:49990
I20260926 02:13:32.823608 31301 tablet_service.cc:1514] Processing DeleteTablet for tablet 32add0586466412bb6af559160f42c91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:32 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:57042
I20260926 02:13:32.824093 31681 tablet_replica.cc:330] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e: stopping tablet replica
I20260926 02:13:32.824272 31683 tablet_replica.cc:330] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692: stopping tablet replica
I20260926 02:13:32.824369 31681 raft_consensus.cc:2229] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:32.824541 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 31097
I20260926 02:13:32.824556 31683 raft_consensus.cc:2229] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:32.824743 31681 raft_consensus.cc:2258] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:32.824951 31682 tablet_replica.cc:330] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035: stopping tablet replica
I20260926 02:13:32.824949 31683 raft_consensus.cc:2258] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:32.825165 31682 raft_consensus.cc:2229] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:13:32.825629 31682 raft_consensus.cc:2258] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:32.826123 31683 ts_tablet_manager.cc:1905] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:32.826895 31682 ts_tablet_manager.cc:1905] T 32add0586466412bb6af559160f42c91 P 2e1dd5d7bb1a4b4fb14c6481c995a035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:32.830366 31681 ts_tablet_manager.cc:1905] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:13:32 dist-test-slave-hllj krb5kdc[30981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388810, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:13:32.848382 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:32.826449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.28.152.129:45125)
0926 02:13:32.834193 (+ 7744us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:32.834198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:32.834214 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:13:32.834262 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:32.834266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:32.834328 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:32.834425 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:32.835768 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:32.836476 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:32.837799 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:32.838196 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:32.838303 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:32.842427 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:32.842443 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:32.842461 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:32.842503 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:32.845448 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:32.846230 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:32.846240 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:32.846242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:32.846386 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:32.846799 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:32.846803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:32.846805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:32.847183 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:32.847462 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:32.847805 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 02:13:32.848040 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7634,"thread_start_us":7519,"threads_started":1}
W20260926 02:13:32.849035 31228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54020
W20260926 02:13:32.849784 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 31097: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:13:32.849970 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 31097
I20260926 02:13:32.873225 31683 ts_tablet_manager.cc:1918] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:13:32.873399 31683 log.cc:1196] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/32add0586466412bb6af559160f42c91
I20260926 02:13:32.873813 31683 ts_tablet_manager.cc:1939] T 32add0586466412bb6af559160f42c91 P a2cedc3998784c2b96e430d920ae7692: Deleting consensus metadata
I20260926 02:13:32.875911 31681 ts_tablet_manager.cc:1918] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:13:32.876053 31681 log.cc:1196] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/32add0586466412bb6af559160f42c91
I20260926 02:13:32.876451 31681 ts_tablet_manager.cc:1939] T 32add0586466412bb6af559160f42c91 P 7ad0967fae494e02a12bb06f9d69ec6e: Deleting consensus metadata
W20260926 02:13:32.887830 31013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.152.129:45125 (error 108)
I20260926 02:13:32.888134 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 31237
I20260926 02:13:32.890165 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:32.889322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45125 (local address 127.0.0.1:54032)
0926 02:13:32.889836 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:32.889972 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":270,"threads_started":1}
W20260926 02:13:32.890372 31013 catalog_manager.cc:4584] TS 2e1dd5d7bb1a4b4fb14c6481c995a035 (127.28.152.129:45125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32add0586466412bb6af559160f42c91: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45125: connect: Connection refused (error 111)
Sep 26 02:13:32 dist-test-slave-hllj krb5kdc[30981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388810, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:13:32.904294 31687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:32.888965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40496 (local address 127.28.152.130:36897)
0926 02:13:32.889401 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:32.889406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:32.889423 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:13:32.889485 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:32.889490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:32.889564 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:32.889718 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:32.891077 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:32.891861 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:32.893152 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:32.893626 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:32.893751 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:32.898108 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:32.898125 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:32.898144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:32.898188 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:32.901269 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:32.902112 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:32.902124 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:32.902127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:32.902272 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:32.902666 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:32.902672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:32.902676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:32.903080 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:32.903363 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:32.903702 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 02:13:32.903936 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":503,"threads_started":1}
W20260926 02:13:32.904996 31367 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40496
W20260926 02:13:32.905823 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 31237: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:13:32.906003 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 31237
I20260926 02:13:32.931665 31011 catalog_manager.cc:4841] TS 7ad0967fae494e02a12bb06f9d69ec6e (127.28.152.131:41993): tablet 32add0586466412bb6af559160f42c91 (table test-table [id=cfb5329ad2134b7e9e59211df1f2262d]) successfully deleted
I20260926 02:13:32.943833 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:32.943346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45125 (local address 127.0.0.1:54038)
0926 02:13:32.943575 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:32.943676 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 02:13:32.945160 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:32.944673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45125 (local address 127.0.0.1:54042)
0926 02:13:32.944837 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:32.944936 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 02:13:32.951560 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 31376
I20260926 02:13:32.951898 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:32.951478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36897 (local address 127.0.0.1:40512)
0926 02:13:32.951647 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:32.951740 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.130:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
Sep 26 02:13:32 dist-test-slave-hllj krb5kdc[30981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388810, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:13:32.967336 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:32.952273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.28.152.131:41993)
0926 02:13:32.952859 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:32.952864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:32.952881 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:13:32.952933 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:32.952938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:32.953003 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:32.953100 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:32.954477 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:32.955256 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:32.956492 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:32.956920 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:32.957021 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:32.961376 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:32.961392 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:32.961411 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:32.961452 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:32.964433 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:32.965313 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:32.965321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:32.965323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:32.965493 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:32.965907 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:32.965912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:32.965915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:32.966299 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:32.966523 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:32.966793 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 02:13:32.966980 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":331,"threads_started":1}
W20260926 02:13:32.968118 31506 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36256
W20260926 02:13:32.968808 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 31376: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:13:32.968956 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 31376
I20260926 02:13:32.984925 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:32.984449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36897 (local address 127.0.0.1:40516)
0926 02:13:32.984671 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:32.984769 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.130:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 02:13:32.986131 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:32.985658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36897 (local address 127.0.0.1:40522)
0926 02:13:32.985891 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:32.985983 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.130:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 02:13:32.988323 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 30994
I20260926 02:13:32.989373 31066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:33.105254 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:33.104502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36897 (local address 127.0.0.1:40534)
0926 02:13:33.104905 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:33.105007 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.130:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284}
I20260926 02:13:33.105942 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 30994
2026-09-26T02:13:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8711 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:13:34 dist-test-slave-hllj krb5kdc[31694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:13:34 dist-test-slave-hllj krb5kdc[31694](info): set up 2 sockets
Sep 26 02:13:34 dist-test-slave-hllj krb5kdc[31694](info): commencing operation
krb5kdc: starting...
W20260926 02:13:35.626664 29282 mini_kdc.cc:121] Time spent starting KDC: real 2.480s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:13:35 dist-test-slave-hllj krb5kdc[31694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:13:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:13:35.820703 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:38615
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:36.155956 31707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:36.156291 31707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:36.156414 31707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:36.163777 31707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:36.163918 31707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:36.163990 31707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:36.164063 31707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:36.164122 31707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:13:36.174273 31707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:36.175647 31707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:36.177016 31707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:36.188905 31712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:36.188937 31713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:36.188959 31715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:36.189973 31707 server_base.cc:1034] running on GCE node
I20260926 02:13:36.190774 31707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:36.192217 31707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:36.193423 31707 hybrid_clock.cc:648] HybridClock initialized: now 1790388816193371 us; error 37 us; skew 500 ppm
I20260926 02:13:36.193773 31707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:36 dist-test-slave-hllj krb5kdc[31694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388816, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:36.204478 31707 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:13:36.206265 31707 webserver.cc:460] Webserver started at http://127.28.152.190:34531/ using document root <none> and password file <none>
I20260926 02:13:36.207018 31707 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:36.207125 31707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:36.207430 31707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:36.210583 31707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "1beaccc1f801457cb445b40f7e00afd7"
format_stamp: "Formatted at 2026-09-26 02:13:36 on dist-test-slave-hllj"
server_key: "ccde62a57f4d5261bd6c11ff686e73d9"
server_key_iv: "72b1709583322f3e3b93ccdc3b07e9ef"
server_key_version: "encryptionkey@0"
I20260926 02:13:36.211323 31707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "1beaccc1f801457cb445b40f7e00afd7"
format_stamp: "Formatted at 2026-09-26 02:13:36 on dist-test-slave-hllj"
server_key: "ccde62a57f4d5261bd6c11ff686e73d9"
server_key_iv: "72b1709583322f3e3b93ccdc3b07e9ef"
server_key_version: "encryptionkey@0"
I20260926 02:13:36.216521 31707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 02:13:36.220202 31722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:36.221688 31707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:13:36.221879 31707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "1beaccc1f801457cb445b40f7e00afd7"
format_stamp: "Formatted at 2026-09-26 02:13:36 on dist-test-slave-hllj"
server_key: "ccde62a57f4d5261bd6c11ff686e73d9"
server_key_iv: "72b1709583322f3e3b93ccdc3b07e9ef"
server_key_version: "encryptionkey@0"
I20260926 02:13:36.222064 31707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:36.235877 31707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:36.239902 31707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:36.240168 31707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:36.257290 31707 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:38615
I20260926 02:13:36.257296 31773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:38615 every 8 connection(s)
I20260926 02:13:36.258774 31707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:36.259313 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 31707
I20260926 02:13:36.259496 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:36.260255 29282 external_mini_cluster.cc:1428] Setting key e6f4488f5567784b97463bd5424459f3
I20260926 02:13:36.262977 31774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:13:36 dist-test-slave-hllj krb5kdc[31694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:36.275931 31774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7: Bootstrap starting.
I20260926 02:13:36.280004 31774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:36.281327 31774 log.cc:824] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:36.284921 31774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7: No bootstrap required, opened a new log
I20260926 02:13:36.285111 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:36.262409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.28.152.190:38615)
0926 02:13:36.263287 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:36.263303 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:36.263712 (+ 409us) server_negotiation.cc:407] Connection header received
0926 02:13:36.265884 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:36.265912 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:36.266438 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:36.266888 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:36.268124 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:36.270141 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:36.271443 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:36.272447 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:36.272651 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:36.276490 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:36.276501 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:36.276521 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:36.276566 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:36.280356 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:36.281119 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:36.281164 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:36.281174 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:36.281412 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:36.281830 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:36.281835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:36.281838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:36.282504 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:36.282781 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:36.283237 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 02:13:36.283701 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":340,"threads_started":1}
I20260926 02:13:36.294003 31774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1beaccc1f801457cb445b40f7e00afd7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38615 } }
I20260926 02:13:36.294421 31774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:36.294497 31774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1beaccc1f801457cb445b40f7e00afd7, State: Initialized, Role: FOLLOWER
I20260926 02:13:36.295042 31774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1beaccc1f801457cb445b40f7e00afd7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38615 } }
I20260926 02:13:36.295249 31774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:36.295346 31774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:36.295473 31774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:36.297864 31774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1beaccc1f801457cb445b40f7e00afd7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38615 } }
I20260926 02:13:36.298431 31774 leader_election.cc:302] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1beaccc1f801457cb445b40f7e00afd7; no voters:
I20260926 02:13:36.298977 31774 leader_election.cc:288] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:36.299115 31779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:36.299692 31779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [term 1 LEADER]: Becoming Leader. State: Replica: 1beaccc1f801457cb445b40f7e00afd7, State: Running, Role: LEADER
I20260926 02:13:36.300232 31779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1beaccc1f801457cb445b40f7e00afd7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38615 } }
I20260926 02:13:36.300968 31774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:36.302794 31780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1beaccc1f801457cb445b40f7e00afd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1beaccc1f801457cb445b40f7e00afd7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38615 } } }
I20260926 02:13:36.303241 31780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:36.304440 31781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1beaccc1f801457cb445b40f7e00afd7. Latest consensus state: current_term: 1 leader_uuid: "1beaccc1f801457cb445b40f7e00afd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1beaccc1f801457cb445b40f7e00afd7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38615 } } }
I20260926 02:13:36.305065 31781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:36.305070 31788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:36.311092 31788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:36.319952 31788 catalog_manager.cc:1347] Generated new cluster ID: 77ebf378d8d44011af9fcc3de8253041
I20260926 02:13:36.320119 31788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:36.359257 31788 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:36.360718 31788 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:36.370996 31788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1beaccc1f801457cb445b40f7e00afd7: Generated new TSK 0
I20260926 02:13:36.371873 31788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:13:36.418919 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:38615
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:36.746728 31800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:36.747014 31800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:36.747115 31800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:36.754415 31800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:36.754546 31800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:36.754671 31800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:13:36.764559 31800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:38615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:36.766033 31800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:36.767398 31800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:36.781278 31805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:36.781174 31806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:36.781503 31808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:36.782239 31800 server_base.cc:1034] running on GCE node
I20260926 02:13:36.782919 31800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:36.783970 31800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:36.785231 31800 hybrid_clock.cc:648] HybridClock initialized: now 1790388816785163 us; error 66 us; skew 500 ppm
I20260926 02:13:36.785578 31800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:36 dist-test-slave-hllj krb5kdc[31694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388816, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:36.788841 31800 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:13:36.790422 31800 webserver.cc:460] Webserver started at http://127.28.152.129:37485/ using document root <none> and password file <none>
I20260926 02:13:36.791124 31800 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:36.791242 31800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:36.791536 31800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:36.794611 31800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "9c28d58e209c470a9c90ca29b5a669f9"
format_stamp: "Formatted at 2026-09-26 02:13:36 on dist-test-slave-hllj"
server_key: "2a708a09c626110051effdb8b8bca579"
server_key_iv: "27510ca84dc1fb6b016bd8a772996430"
server_key_version: "encryptionkey@0"
I20260926 02:13:36.795419 31800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "9c28d58e209c470a9c90ca29b5a669f9"
format_stamp: "Formatted at 2026-09-26 02:13:36 on dist-test-slave-hllj"
server_key: "2a708a09c626110051effdb8b8bca579"
server_key_iv: "27510ca84dc1fb6b016bd8a772996430"
server_key_version: "encryptionkey@0"
I20260926 02:13:36.801295 31800 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 02:13:36.805238 31815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:36.806766 31800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:36.806946 31800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "9c28d58e209c470a9c90ca29b5a669f9"
format_stamp: "Formatted at 2026-09-26 02:13:36 on dist-test-slave-hllj"
server_key: "2a708a09c626110051effdb8b8bca579"
server_key_iv: "27510ca84dc1fb6b016bd8a772996430"
server_key_version: "encryptionkey@0"
I20260926 02:13:36.807158 31800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:36.822551 31800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:36.826891 31800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:36.827163 31800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:36.828130 31800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:36.829511 31800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:36.829602 31800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:36.829722 31800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:36.829778 31800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:36.859468 31800 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:34915
I20260926 02:13:36.859505 31927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:34915 every 8 connection(s)
I20260926 02:13:36.860733 31800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:13:36.868927 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 31800
I20260926 02:13:36.869107 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:13:36.869921 29282 external_mini_cluster.cc:1428] Setting key 005aa023ec0c3b2a7bc5d79292968f53
Sep 26 02:13:36 dist-test-slave-hllj krb5kdc[31694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388816, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:36.881198 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:36.863246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:56449 (local address 127.28.152.190:38615)
0926 02:13:36.863646 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:36.863651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:36.864724 (+ 1073us) server_negotiation.cc:407] Connection header received
0926 02:13:36.866506 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:36.866510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:36.866595 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:36.866692 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:36.868165 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:36.868939 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:36.871137 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:36.871780 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:36.871909 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:36.875753 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:36.875758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:36.875762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:36.875794 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:36.878150 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:36.878904 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:36.878908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:36.878911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:36.879030 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:36.879448 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:36.879452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:36.879454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:36.879734 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:36.879867 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:36.880575 (+ 708us) server_negotiation.cc:299] Negotiation successful
0926 02:13:36.880808 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":207,"threads_started":1}
I20260926 02:13:36.882346 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:36.863837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:38615 (local address 127.28.152.129:56449)
0926 02:13:36.864523 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:36.864566 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:36.866231 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:36.866918 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:36.866926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:36.867419 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:36.867993 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:36.868011 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:36.869109 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:36.869114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:36.870984 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:36.870997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:36.871991 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:36.871998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:36.872288 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:36.872996 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:36.873023 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:36.875521 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:36.878305 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:36.878316 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:36.878337 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:36.878775 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:36.879131 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:36.879135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:36.879140 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:36.879337 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:36.879859 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:36.879866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:36.880222 (+ 356us) client_negotiation.cc:769] Sending connection context
0926 02:13:36.880465 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 02:13:36.881042 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":260,"threads_started":1}
I20260926 02:13:36.884006 31928 heartbeater.cc:347] Connected to a master server at 127.28.152.190:38615
I20260926 02:13:36.884352 31928 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:36.885282 31928 heartbeater.cc:511] Master 127.28.152.190:38615 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
I20260926 02:13:36.887593 31739 ts_manager.cc:194] Registered new tserver with Master: 9c28d58e209c470a9c90ca29b5a669f9 (127.28.152.129:34915)
I20260926 02:13:36.889191 31739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:56449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:13:36.911934 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:38615
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:37.232357 31934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:37.232666 31934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:37.232815 31934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:37.240537 31934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:37.240665 31934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:37.240823 31934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:13:37.250967 31934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:38615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:37.252285 31934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:37.253647 31934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:37.266772 31940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:37.266947 31939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:37.267201 31942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:37.267655 31934 server_base.cc:1034] running on GCE node
I20260926 02:13:37.268263 31934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:37.269263 31934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:37.270478 31934 hybrid_clock.cc:648] HybridClock initialized: now 1790388817270414 us; error 54 us; skew 500 ppm
I20260926 02:13:37.270802 31934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:37 dist-test-slave-hllj krb5kdc[31694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388817, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:37.273933 31934 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:13:37.275413 31934 webserver.cc:460] Webserver started at http://127.28.152.130:33393/ using document root <none> and password file <none>
I20260926 02:13:37.276098 31934 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:37.276209 31934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:37.276492 31934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:37.279446 31934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "49aaf950fec043c496d3af94e1b3f897"
format_stamp: "Formatted at 2026-09-26 02:13:37 on dist-test-slave-hllj"
server_key: "b27ccdc2043448adc4bab6d258efa74a"
server_key_iv: "86057bf7f5980c8fc1884df25367f677"
server_key_version: "encryptionkey@0"
I20260926 02:13:37.280221 31934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "49aaf950fec043c496d3af94e1b3f897"
format_stamp: "Formatted at 2026-09-26 02:13:37 on dist-test-slave-hllj"
server_key: "b27ccdc2043448adc4bab6d258efa74a"
server_key_iv: "86057bf7f5980c8fc1884df25367f677"
server_key_version: "encryptionkey@0"
I20260926 02:13:37.285679 31934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:13:37.289289 31949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:37.290669 31934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:37.290828 31934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "49aaf950fec043c496d3af94e1b3f897"
format_stamp: "Formatted at 2026-09-26 02:13:37 on dist-test-slave-hllj"
server_key: "b27ccdc2043448adc4bab6d258efa74a"
server_key_iv: "86057bf7f5980c8fc1884df25367f677"
server_key_version: "encryptionkey@0"
I20260926 02:13:37.291015 31934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:37.301882 31934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:37.306164 31934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:37.306438 31934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:37.307379 31934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:37.308706 31934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:37.308782 31934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:37.308895 31934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:37.308946 31934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:37.339397 31934 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:42479
I20260926 02:13:37.339452 32061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:42479 every 8 connection(s)
I20260926 02:13:37.340672 31934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:13:37.349925 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 31934
I20260926 02:13:37.350077 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:13:37.350510 29282 external_mini_cluster.cc:1428] Setting key 9856e7e82e1e6287ee909cf872c58d60
Sep 26 02:13:37 dist-test-slave-hllj krb5kdc[31694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388817, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:37.361249 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:37.343186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:60641 (local address 127.28.152.190:38615)
0926 02:13:37.343373 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:37.343378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:37.344552 (+ 1174us) server_negotiation.cc:407] Connection header received
0926 02:13:37.346422 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:37.346427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:37.346515 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:37.346590 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:37.348090 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:37.348798 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:37.350498 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:37.351117 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:37.351222 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:37.355487 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:37.355493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:37.355497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:37.355534 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:37.358110 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:37.358967 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:37.358974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:37.358978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:37.359134 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:37.359509 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:37.359514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:37.359517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:37.359856 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:37.359987 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:37.360670 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 02:13:37.360899 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:13:37.362129 32064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:37.343655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:38615 (local address 127.28.152.130:60641)
0926 02:13:37.344385 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:37.344424 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:37.346124 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:37.346799 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:37.346807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:37.347356 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:37.347951 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:37.347968 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:37.348953 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:37.348957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:37.350341 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:37.350354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:37.351369 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:37.351378 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:37.351642 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:37.352439 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:37.352474 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:37.355196 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:37.358287 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:37.358298 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:37.358320 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:37.358823 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:37.359230 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:37.359233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:37.359235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:37.359411 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:37.359966 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:37.359971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:37.360301 (+ 330us) client_negotiation.cc:769] Sending connection context
0926 02:13:37.360529 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 02:13:37.360942 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":279,"threads_started":1}
I20260926 02:13:37.363616 32062 heartbeater.cc:347] Connected to a master server at 127.28.152.190:38615
I20260926 02:13:37.363929 32062 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:37.364671 32062 heartbeater.cc:511] Master 127.28.152.190:38615 requested a full tablet report, sending...
I20260926 02:13:37.366218 31739 ts_manager.cc:194] Registered new tserver with Master: 49aaf950fec043c496d3af94e1b3f897 (127.28.152.130:42479)
I20260926 02:13:37.367003 31739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:60641
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:13:37.393368 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:38615
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:37.716756 32067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:37.717075 32067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:37.717231 32067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:37.724650 32067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:37.724808 32067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:37.724954 32067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:13:37.735096 32067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:38615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:37.736696 32067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:37.738374 32067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:37.751636 32073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:37.751586 32072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:37.751608 32075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:37.752552 32067 server_base.cc:1034] running on GCE node
I20260926 02:13:37.753216 32067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:37.754235 32067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:37.755448 32067 hybrid_clock.cc:648] HybridClock initialized: now 1790388817755394 us; error 52 us; skew 500 ppm
I20260926 02:13:37.755748 32067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:37 dist-test-slave-hllj krb5kdc[31694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388817, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:37.758790 32067 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:13:37.760281 32067 webserver.cc:460] Webserver started at http://127.28.152.131:37329/ using document root <none> and password file <none>
I20260926 02:13:37.760980 32067 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:37.761101 32067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:37.761444 32067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:37.764387 32067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "6c143fcf9d1f4f91be852fe8c0c75f5a"
format_stamp: "Formatted at 2026-09-26 02:13:37 on dist-test-slave-hllj"
server_key: "1a250f725c76e5b1bc3dc4cc773bc0d7"
server_key_iv: "d42c44d0659b45d7dcd19b3f05a9d8f8"
server_key_version: "encryptionkey@0"
I20260926 02:13:37.765198 32067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "6c143fcf9d1f4f91be852fe8c0c75f5a"
format_stamp: "Formatted at 2026-09-26 02:13:37 on dist-test-slave-hllj"
server_key: "1a250f725c76e5b1bc3dc4cc773bc0d7"
server_key_iv: "d42c44d0659b45d7dcd19b3f05a9d8f8"
server_key_version: "encryptionkey@0"
I20260926 02:13:37.770606 32067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 02:13:37.774224 32082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:37.775648 32067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 02:13:37.775820 32067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "6c143fcf9d1f4f91be852fe8c0c75f5a"
format_stamp: "Formatted at 2026-09-26 02:13:37 on dist-test-slave-hllj"
server_key: "1a250f725c76e5b1bc3dc4cc773bc0d7"
server_key_iv: "d42c44d0659b45d7dcd19b3f05a9d8f8"
server_key_version: "encryptionkey@0"
I20260926 02:13:37.776077 32067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:37.792372 32067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:37.796916 32067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:37.797236 32067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:37.798282 32067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:37.799767 32067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:37.799842 32067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:37.799933 32067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:37.799993 32067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:37.830297 32067 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:33899
I20260926 02:13:37.830335 32194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:33899 every 8 connection(s)
I20260926 02:13:37.831570 32067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:13:37.841053 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 32067
I20260926 02:13:37.841224 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:13:37.841730 29282 external_mini_cluster.cc:1428] Setting key 300f2558765ccf9b9617eee65d11eafd
Sep 26 02:13:37 dist-test-slave-hllj krb5kdc[31694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388817, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:37.851523 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:37.834147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:49443 (local address 127.28.152.190:38615)
0926 02:13:37.834324 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:37.834328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:37.835595 (+ 1267us) server_negotiation.cc:407] Connection header received
0926 02:13:37.837440 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:37.837444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:37.837514 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:37.837590 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:37.839122 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:37.839859 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:37.841566 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:37.842162 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:37.842310 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:37.846319 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:37.846324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:37.846327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:37.846361 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:37.848593 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:37.849333 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:37.849337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:37.849340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:37.849451 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:37.849859 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:37.849862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:37.849865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:37.850138 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:37.850275 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:37.850965 (+ 690us) server_negotiation.cc:299] Negotiation successful
0926 02:13:37.851179 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:13:37.852491 32197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:37.834682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:38615 (local address 127.28.152.131:49443)
0926 02:13:37.835416 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:37.835457 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:37.837109 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:37.837799 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:37.837807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:37.838361 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:37.838960 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:37.838979 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:37.839992 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:37.839996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:37.841407 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:37.841419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:37.842365 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:37.842370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:37.842651 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:37.843372 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:37.843402 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:37.846114 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:37.848736 (+ 2622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:37.848746 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:37.848773 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:37.849219 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:37.849548 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:37.849554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:37.849558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:37.849742 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:37.850257 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:37.850265 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:37.850616 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 02:13:37.850861 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 02:13:37.851393 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":276,"threads_started":1}
I20260926 02:13:37.853863 32195 heartbeater.cc:347] Connected to a master server at 127.28.152.190:38615
I20260926 02:13:37.854166 32195 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:37.854945 32195 heartbeater.cc:511] Master 127.28.152.190:38615 requested a full tablet report, sending...
I20260926 02:13:37.856604 31739 ts_manager.cc:194] Registered new tserver with Master: 6c143fcf9d1f4f91be852fe8c0c75f5a (127.28.152.131:33899)
I20260926 02:13:37.857633 31739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:49443
I20260926 02:13:37.858144 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 02:13:37.888410 31930 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:13:37.886026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.28.152.190:38615)
0926 02:13:37.886221 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:37.886227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:37.886601 (+ 374us) server_negotiation.cc:407] Connection header received
0926 02:13:37.886777 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:37.886781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:37.886858 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:37.886941 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:37.888215 (+ 1274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37140: BlockingRecv error: recv got EOF from 127.0.0.1:37140 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:13:37.891557 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 31800
I20260926 02:13:37.892468 31928 heartbeater.cc:503] Master 127.28.152.190:38615 was elected leader, sending a full tablet report...
W20260926 02:13:37.894526 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 31800: Not authorized: Client connection negotiation failed: client connection to 127.28.152.129:34915: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:13:37.894726 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 31800
I20260926 02:13:37.906816 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 31934
W20260926 02:13:37.909504 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 31934: Not authorized: Client connection negotiation failed: client connection to 127.28.152.130:42479: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:13:37.909632 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 31934
I20260926 02:13:37.921852 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 32067
W20260926 02:13:37.924880 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 32067: Not authorized: Client connection negotiation failed: client connection to 127.28.152.131:33899: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:13:37.925057 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 32067
I20260926 02:13:37.937791 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 31707
I20260926 02:13:37.939076 31769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:38.039403 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 31707
2026-09-26T02:13:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4929 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:13:38 dist-test-slave-hllj krb5kdc[32212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:13:38 dist-test-slave-hllj krb5kdc[32212](info): set up 2 sockets
Sep 26 02:13:38 dist-test-slave-hllj krb5kdc[32212](info): commencing operation
krb5kdc: starting...
W20260926 02:13:39.034662 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:13:39 dist-test-slave-hllj krb5kdc[32212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:13:39.237231 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:42573
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:42573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:39.560286 32225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:39.560597 32225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:39.560696 32225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:39.568004 32225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:39.568130 32225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:39.568176 32225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:39.568230 32225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:39.568277 32225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:13:39.578313 32225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:42573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:42573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:39.579614 32225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:39.581027 32225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:39.593102 32231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:39.593199 32233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:39.593195 32230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:39.594092 32225 server_base.cc:1034] running on GCE node
I20260926 02:13:39.594991 32225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:39.596536 32225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:39.597751 32225 hybrid_clock.cc:648] HybridClock initialized: now 1790388819597689 us; error 57 us; skew 500 ppm
I20260926 02:13:39.598117 32225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:39 dist-test-slave-hllj krb5kdc[32212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388819, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:39.601297 32225 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:13:39.602837 32225 webserver.cc:460] Webserver started at http://127.28.152.190:39847/ using document root <none> and password file <none>
I20260926 02:13:39.603515 32225 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:39.603614 32225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:39.603978 32225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:39.607153 32225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "6e4c70a12d6641d99c999f9bd2f5bf41"
format_stamp: "Formatted at 2026-09-26 02:13:39 on dist-test-slave-hllj"
server_key: "60d45652210ae98749005caa9c23f479"
server_key_iv: "5e70cfb36be34d890910de59fe22a017"
server_key_version: "encryptionkey@0"
I20260926 02:13:39.607925 32225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "6e4c70a12d6641d99c999f9bd2f5bf41"
format_stamp: "Formatted at 2026-09-26 02:13:39 on dist-test-slave-hllj"
server_key: "60d45652210ae98749005caa9c23f479"
server_key_iv: "5e70cfb36be34d890910de59fe22a017"
server_key_version: "encryptionkey@0"
I20260926 02:13:39.613525 32225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:13:39.617216 32240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:39.618690 32225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:39.618856 32225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "6e4c70a12d6641d99c999f9bd2f5bf41"
format_stamp: "Formatted at 2026-09-26 02:13:39 on dist-test-slave-hllj"
server_key: "60d45652210ae98749005caa9c23f479"
server_key_iv: "5e70cfb36be34d890910de59fe22a017"
server_key_version: "encryptionkey@0"
I20260926 02:13:39.619050 32225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:39.637984 32225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:39.642465 32225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:39.642740 32225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:39.660405 32225 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:42573
I20260926 02:13:39.660427 32291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:42573 every 8 connection(s)
I20260926 02:13:39.661872 32225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:39.665827 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 32225
I20260926 02:13:39.665918 32292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:39.666038 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:39.666654 29282 external_mini_cluster.cc:1428] Setting key 4afe7c780b20c3ad632a7680b609de53
I20260926 02:13:39.681316 32292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41: Bootstrap starting.
Sep 26 02:13:39 dist-test-slave-hllj krb5kdc[32212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:39.684931 32292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:39.686318 32292 log.cc:824] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:39.689797 32292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41: No bootstrap required, opened a new log
I20260926 02:13:39.691883 32295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:39.668625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.28.152.190:42573)
0926 02:13:39.669555 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:39.669572 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:39.669628 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:13:39.671297 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:39.671331 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:39.671948 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:39.672458 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:39.673928 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:39.675849 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:39.677716 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:39.678488 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:39.678642 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:39.683501 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:39.683512 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:39.683528 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:39.683579 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:39.687062 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:39.687944 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:39.687954 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:39.687963 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:39.688158 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:39.688531 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:39.688536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:39.688539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:39.689329 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:39.689622 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:39.690042 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 02:13:39.690521 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":407,"threads_started":1}
I20260926 02:13:39.695796 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:13:39.698203 32292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4c70a12d6641d99c999f9bd2f5bf41" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42573 } }
I20260926 02:13:39.698606 32292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:39.698704 32292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4c70a12d6641d99c999f9bd2f5bf41, State: Initialized, Role: FOLLOWER
I20260926 02:13:39.699294 32292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4c70a12d6641d99c999f9bd2f5bf41" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42573 } }
I20260926 02:13:39.699501 32292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:39.699636 32292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:39.699786 32292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:39.701993 32292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4c70a12d6641d99c999f9bd2f5bf41" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42573 } }
I20260926 02:13:39.702569 32292 leader_election.cc:302] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4c70a12d6641d99c999f9bd2f5bf41; no voters:
I20260926 02:13:39.703197 32292 leader_election.cc:288] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:39.703322 32298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:39.704485 32298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [term 1 LEADER]: Becoming Leader. State: Replica: 6e4c70a12d6641d99c999f9bd2f5bf41, State: Running, Role: LEADER
I20260926 02:13:39.705001 32298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4c70a12d6641d99c999f9bd2f5bf41" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42573 } }
I20260926 02:13:39.705554 32292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:39.707789 32300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e4c70a12d6641d99c999f9bd2f5bf41. Latest consensus state: current_term: 1 leader_uuid: "6e4c70a12d6641d99c999f9bd2f5bf41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4c70a12d6641d99c999f9bd2f5bf41" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42573 } } }
I20260926 02:13:39.707800 32299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e4c70a12d6641d99c999f9bd2f5bf41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4c70a12d6641d99c999f9bd2f5bf41" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42573 } } }
I20260926 02:13:39.708242 32300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:39.708329 32299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:39.720122 32303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:39.726250 32303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:39.745568 32303 catalog_manager.cc:1347] Generated new cluster ID: ed734ad4120b43c1bab54316d54fe345
I20260926 02:13:39.745739 32303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:39.768289 32295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:39.759108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.28.152.190:42573)
0926 02:13:39.759289 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:39.759294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:39.759931 (+ 637us) server_negotiation.cc:407] Connection header received
0926 02:13:39.760115 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:39.760121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:39.760214 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:39.760328 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:39.761484 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:39.762607 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:39.764376 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:39.765046 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:39.765249 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:39.766488 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:39.766495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:39.766500 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:39.767455 (+ 955us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:39.767518 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:39.767555 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:39.767672 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:39.767812 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 02:13:39.767918 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:13:39.772534 32303 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:39.774477 32303 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:39.784883 32303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e4c70a12d6641d99c999f9bd2f5bf41: Generated new TSK 0
I20260926 02:13:39.786170 32303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:13:39.798661 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 32225
I20260926 02:13:39.799939 32287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:39.890625 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 32225
2026-09-26T02:13:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1846 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:13:39 dist-test-slave-hllj krb5kdc[32326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:13:39 dist-test-slave-hllj krb5kdc[32326](info): set up 2 sockets
Sep 26 02:13:39 dist-test-slave-hllj krb5kdc[32326](info): commencing operation
krb5kdc: starting...
W20260926 02:13:40.894729 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:13:41 dist-test-slave-hllj krb5kdc[32326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:13:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:13:41.086582 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:44971
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:41.411194 32339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:41.411504 32339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:41.411599 32339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:41.418995 32339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:41.419126 32339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:41.419173 32339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:41.419226 32339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:41.419269 32339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:13:41.429425 32339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:41.430773 32339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:41.432094 32339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:41.444372 32344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:41.444299 32347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:41.444351 32345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:41.446028 32339 server_base.cc:1034] running on GCE node
I20260926 02:13:41.447017 32339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:41.448623 32339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:41.449823 32339 hybrid_clock.cc:648] HybridClock initialized: now 1790388821449740 us; error 49 us; skew 500 ppm
I20260926 02:13:41.450343 32339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:41 dist-test-slave-hllj krb5kdc[32326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388821, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:41.466151 32339 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:13:41.468015 32339 webserver.cc:460] Webserver started at http://127.28.152.190:36361/ using document root <none> and password file <none>
I20260926 02:13:41.468788 32339 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:41.468917 32339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:41.469249 32339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:41.472393 32339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "bf7e14ecaf2043dc859bd50ac00a0427"
format_stamp: "Formatted at 2026-09-26 02:13:41 on dist-test-slave-hllj"
server_key: "f55dded16f65aac4587f66466420a224"
server_key_iv: "9decf289cea3694e683bc61222d9ef67"
server_key_version: "encryptionkey@0"
I20260926 02:13:41.473244 32339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "bf7e14ecaf2043dc859bd50ac00a0427"
format_stamp: "Formatted at 2026-09-26 02:13:41 on dist-test-slave-hllj"
server_key: "f55dded16f65aac4587f66466420a224"
server_key_iv: "9decf289cea3694e683bc61222d9ef67"
server_key_version: "encryptionkey@0"
I20260926 02:13:41.478974 32339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:13:41.482892 32354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:41.484462 32339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:13:41.484644 32339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "bf7e14ecaf2043dc859bd50ac00a0427"
format_stamp: "Formatted at 2026-09-26 02:13:41 on dist-test-slave-hllj"
server_key: "f55dded16f65aac4587f66466420a224"
server_key_iv: "9decf289cea3694e683bc61222d9ef67"
server_key_version: "encryptionkey@0"
I20260926 02:13:41.484833 32339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:41.508077 32339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:41.512516 32339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:41.512801 32339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:41.531533 32339 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:44971
I20260926 02:13:41.531556 32405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:44971 every 8 connection(s)
I20260926 02:13:41.532948 32339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:41.535207 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 32339
I20260926 02:13:41.535424 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:41.535998 29282 external_mini_cluster.cc:1428] Setting key df77f4fb454f80ee72554c6c4e0a880e
I20260926 02:13:41.537322 32406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:13:41 dist-test-slave-hllj krb5kdc[32326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:41.551694 32406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427: Bootstrap starting.
I20260926 02:13:41.555840 32406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:41.557119 32406 log.cc:824] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:41.560946 32406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427: No bootstrap required, opened a new log
I20260926 02:13:41.561542 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:41.538192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.28.152.190:44971)
0926 02:13:41.539026 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:41.539043 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:41.539094 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:13:41.540885 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:41.540918 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:41.541553 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:41.542081 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:41.543373 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:41.545437 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:41.546817 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:41.547898 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:41.548143 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:41.552410 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:41.552422 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:41.552445 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:41.552495 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:41.556437 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:41.557350 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:41.557359 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:41.557368 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:41.557590 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:41.558127 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:41.558131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:41.558133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:41.558850 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:41.559172 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:41.559582 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 02:13:41.560049 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":339,"threads_started":1}
I20260926 02:13:41.569161 32406 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7e14ecaf2043dc859bd50ac00a0427" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44971 } }
I20260926 02:13:41.569545 32406 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:41.569620 32406 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7e14ecaf2043dc859bd50ac00a0427, State: Initialized, Role: FOLLOWER
I20260926 02:13:41.570142 32406 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7e14ecaf2043dc859bd50ac00a0427" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44971 } }
I20260926 02:13:41.570358 32406 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:41.570457 32406 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:41.570583 32406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:41.573009 32406 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7e14ecaf2043dc859bd50ac00a0427" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44971 } }
I20260926 02:13:41.573530 32406 leader_election.cc:302] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf7e14ecaf2043dc859bd50ac00a0427; no voters:
I20260926 02:13:41.574057 32406 leader_election.cc:288] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:41.574196 32411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:41.574808 32411 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [term 1 LEADER]: Becoming Leader. State: Replica: bf7e14ecaf2043dc859bd50ac00a0427, State: Running, Role: LEADER
I20260926 02:13:41.575325 32411 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7e14ecaf2043dc859bd50ac00a0427" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44971 } }
I20260926 02:13:41.576058 32406 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:41.578579 32413 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf7e14ecaf2043dc859bd50ac00a0427. Latest consensus state: current_term: 1 leader_uuid: "bf7e14ecaf2043dc859bd50ac00a0427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7e14ecaf2043dc859bd50ac00a0427" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44971 } } }
I20260926 02:13:41.578967 32413 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:41.578826 32412 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf7e14ecaf2043dc859bd50ac00a0427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7e14ecaf2043dc859bd50ac00a0427" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44971 } } }
I20260926 02:13:41.579169 32412 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:41.582083 32420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:41.586936 32420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:41.596187 32420 catalog_manager.cc:1347] Generated new cluster ID: a5d01f7775e2417bb29b7b56db6227be
I20260926 02:13:41.596320 32420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:41.608410 32420 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:41.609896 32420 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:41.620179 32420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf7e14ecaf2043dc859bd50ac00a0427: Generated new TSK 0
I20260926 02:13:41.621289 32420 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:13:41.676528 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44971
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:42.013279 32432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:42.013593 32432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:42.013700 32432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:42.021098 32432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:42.021253 32432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:42.021384 32432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:13:42.031221 32432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:42.032509 32432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:42.033861 32432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:42.046808 32437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:42.046733 32438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:42.046757 32440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:42.047715 32432 server_base.cc:1034] running on GCE node
I20260926 02:13:42.048377 32432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:42.049489 32432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:42.050743 32432 hybrid_clock.cc:648] HybridClock initialized: now 1790388822050683 us; error 53 us; skew 500 ppm
I20260926 02:13:42.051090 32432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:42 dist-test-slave-hllj krb5kdc[32326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388822, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:42.054512 32432 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:13:42.056165 32432 webserver.cc:460] Webserver started at http://127.28.152.129:35791/ using document root <none> and password file <none>
I20260926 02:13:42.056878 32432 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:42.056984 32432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:42.057327 32432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:42.060604 32432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "a1c23e02699b4db09d89209c63a3e317"
format_stamp: "Formatted at 2026-09-26 02:13:42 on dist-test-slave-hllj"
server_key: "683650c30efd83028b7b7bd320e22696"
server_key_iv: "f158a6a3d99dd72faa2ad60283e88e13"
server_key_version: "encryptionkey@0"
I20260926 02:13:42.061498 32432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "a1c23e02699b4db09d89209c63a3e317"
format_stamp: "Formatted at 2026-09-26 02:13:42 on dist-test-slave-hllj"
server_key: "683650c30efd83028b7b7bd320e22696"
server_key_iv: "f158a6a3d99dd72faa2ad60283e88e13"
server_key_version: "encryptionkey@0"
I20260926 02:13:42.066854 32432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:13:42.070592 32447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:42.072129 32432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:42.072309 32432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "a1c23e02699b4db09d89209c63a3e317"
format_stamp: "Formatted at 2026-09-26 02:13:42 on dist-test-slave-hllj"
server_key: "683650c30efd83028b7b7bd320e22696"
server_key_iv: "f158a6a3d99dd72faa2ad60283e88e13"
server_key_version: "encryptionkey@0"
I20260926 02:13:42.072513 32432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:42.091689 32432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:42.095933 32432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:42.096217 32432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:42.097203 32432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:42.098632 32432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:42.098716 32432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:42.098850 32432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:42.098903 32432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:42.129153 32432 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:45671
I20260926 02:13:42.129258 32559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:45671 every 8 connection(s)
I20260926 02:13:42.130472 32432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:13:42.139007 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 32432
I20260926 02:13:42.139228 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:13:42.140029 29282 external_mini_cluster.cc:1428] Setting key 421c7ae924d7a928a15151f90ac80cbc
Sep 26 02:13:42 dist-test-slave-hllj krb5kdc[32326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388822, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:42.151616 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:42.133049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:35345 (local address 127.28.152.190:44971)
0926 02:13:42.133537 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:42.133544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:42.134653 (+ 1109us) server_negotiation.cc:407] Connection header received
0926 02:13:42.136439 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:42.136443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:42.136511 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:42.136608 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:42.138279 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:42.138986 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:42.141229 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:42.141649 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:42.141772 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:42.145452 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:42.145458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:42.145464 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:42.145506 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:42.148220 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:42.149179 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:42.149184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:42.149188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:42.149312 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:42.149853 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:42.149858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:42.149860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:42.150149 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:42.150268 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:42.151158 (+ 890us) server_negotiation.cc:299] Negotiation successful
0926 02:13:42.151347 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":263,"threads_started":1}
I20260926 02:13:42.152976 32563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:42.133722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:44971 (local address 127.28.152.129:35345)
0926 02:13:42.134485 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:42.134526 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:42.136160 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:42.136853 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:42.136863 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:42.137477 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:42.138092 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:42.138111 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:42.139229 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:42.139235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:42.141033 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:42.141049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:42.141897 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:42.141906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:42.142176 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:42.142818 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:42.142846 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:42.145167 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:42.148401 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:42.148412 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:42.148430 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:42.149027 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:42.149411 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:42.149416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:42.149418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:42.149660 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:42.150272 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:42.150282 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:42.150736 (+ 454us) client_negotiation.cc:769] Sending connection context
0926 02:13:42.151036 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 02:13:42.151575 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":295,"threads_started":1}
I20260926 02:13:42.154620 32560 heartbeater.cc:347] Connected to a master server at 127.28.152.190:44971
I20260926 02:13:42.154958 32560 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:42.155890 32560 heartbeater.cc:511] Master 127.28.152.190:44971 requested a full tablet report, sending...
I20260926 02:13:42.158284 32371 ts_manager.cc:194] Registered new tserver with Master: a1c23e02699b4db09d89209c63a3e317 (127.28.152.129:45671)
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
I20260926 02:13:42.159813 32371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:35345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:13:42.185556 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44971
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:42.512818 32566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:42.513152 32566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:42.513289 32566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:42.520690 32566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:42.520817 32566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:42.520980 32566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:13:42.531126 32566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:42.532399 32566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:42.533702 32566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:42.546945 32571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:42.546893 32572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:42.547225 32574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:42.547669 32566 server_base.cc:1034] running on GCE node
I20260926 02:13:42.548352 32566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:42.549423 32566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:42.550680 32566 hybrid_clock.cc:648] HybridClock initialized: now 1790388822550624 us; error 50 us; skew 500 ppm
I20260926 02:13:42.551044 32566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:42 dist-test-slave-hllj krb5kdc[32326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388822, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:42.554553 32566 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:13:42.556141 32566 webserver.cc:460] Webserver started at http://127.28.152.130:44767/ using document root <none> and password file <none>
I20260926 02:13:42.556877 32566 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:42.557003 32566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:42.557360 32566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:42.560418 32566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "71cb2b330de34d6a9a3ec1267c39be6f"
format_stamp: "Formatted at 2026-09-26 02:13:42 on dist-test-slave-hllj"
server_key: "04d62e8d371475455d6aa213465643e3"
server_key_iv: "aadc5f9a6e8e689772ae60efb9ab09bf"
server_key_version: "encryptionkey@0"
I20260926 02:13:42.561260 32566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "71cb2b330de34d6a9a3ec1267c39be6f"
format_stamp: "Formatted at 2026-09-26 02:13:42 on dist-test-slave-hllj"
server_key: "04d62e8d371475455d6aa213465643e3"
server_key_iv: "aadc5f9a6e8e689772ae60efb9ab09bf"
server_key_version: "encryptionkey@0"
I20260926 02:13:42.566753 32566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 02:13:42.570559 32581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:42.572248 32566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:42.572433 32566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "71cb2b330de34d6a9a3ec1267c39be6f"
format_stamp: "Formatted at 2026-09-26 02:13:42 on dist-test-slave-hllj"
server_key: "04d62e8d371475455d6aa213465643e3"
server_key_iv: "aadc5f9a6e8e689772ae60efb9ab09bf"
server_key_version: "encryptionkey@0"
I20260926 02:13:42.572633 32566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:42.595665 32566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:42.600003 32566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:42.600328 32566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:42.601377 32566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:42.602713 32566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:42.602793 32566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:42.602905 32566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:42.602957 32566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:42.633558 32566 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:46075
I20260926 02:13:42.633627 32693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:46075 every 8 connection(s)
I20260926 02:13:42.634853 32566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:13:42.644064 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 32566
I20260926 02:13:42.644237 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:13:42.644631 29282 external_mini_cluster.cc:1428] Setting key 2efc04a71d3e5f6f774088396c7c69c9
Sep 26 02:13:42 dist-test-slave-hllj krb5kdc[32326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388822, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:42.655045 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:42.637437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:40515 (local address 127.28.152.190:44971)
0926 02:13:42.637608 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:42.637613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:42.638879 (+ 1266us) server_negotiation.cc:407] Connection header received
0926 02:13:42.640732 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:42.640738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:42.640828 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:42.640920 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:42.642502 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:42.643206 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:42.644861 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:42.645327 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:42.645442 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:42.649282 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:42.649288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:42.649291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:42.649327 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:42.651809 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:42.652597 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:42.652601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:42.652604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:42.652742 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:42.653219 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:42.653225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:42.653228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:42.653503 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:42.653643 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:42.654438 (+ 795us) server_negotiation.cc:299] Negotiation successful
0926 02:13:42.654684 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:13:42.656190 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:42.637988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:44971 (local address 127.28.152.130:40515)
0926 02:13:42.638704 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:42.638758 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:42.640440 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:42.641210 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:42.641219 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:42.641735 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:42.642330 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:42.642351 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:42.643359 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:42.643363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:42.644714 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:42.644732 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:42.645551 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:42.645558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:42.645814 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:42.646512 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:42.646537 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:42.648986 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:42.651974 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:42.651984 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:42.652002 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:42.652476 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:42.652864 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:42.652868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:42.652872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:42.653092 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:42.653631 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:42.653641 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:42.654042 (+ 401us) client_negotiation.cc:769] Sending connection context
0926 02:13:42.654306 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 02:13:42.654925 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":281,"threads_started":1}
I20260926 02:13:42.657884 32694 heartbeater.cc:347] Connected to a master server at 127.28.152.190:44971
I20260926 02:13:42.658274 32694 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:42.659332 32694 heartbeater.cc:511] Master 127.28.152.190:44971 requested a full tablet report, sending...
I20260926 02:13:42.661267 32371 ts_manager.cc:194] Registered new tserver with Master: 71cb2b330de34d6a9a3ec1267c39be6f (127.28.152.130:46075)
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:13:42.662395 32371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:40515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:13:42.688176 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44971
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:43.013795 32699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:43.014073 32699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:43.014174 32699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:43.021363 32699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:43.021482 32699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:43.021603 32699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:13:43.031271 32699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:43.032546 32699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:43.033918 32699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:43.046517 32704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:43.046548 32705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:43.046622 32707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:43.047322 32699 server_base.cc:1034] running on GCE node
I20260926 02:13:43.047932 32699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:43.048872 32699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:43.050089 32699 hybrid_clock.cc:648] HybridClock initialized: now 1790388823050014 us; error 66 us; skew 500 ppm
I20260926 02:13:43.050428 32699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:43 dist-test-slave-hllj krb5kdc[32326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388823, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:43.053468 32699 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:13:43.054838 32699 webserver.cc:460] Webserver started at http://127.28.152.131:42409/ using document root <none> and password file <none>
I20260926 02:13:43.055508 32699 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:43.055606 32699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:43.055904 32699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:43.058948 32699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "130ff8aac51a4b74a3e42e5cd2d72259"
format_stamp: "Formatted at 2026-09-26 02:13:43 on dist-test-slave-hllj"
server_key: "5beecc39e491681071f52d3a14944bb0"
server_key_iv: "a48507d303da536b726a89dc8f510c51"
server_key_version: "encryptionkey@0"
I20260926 02:13:43.059702 32699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "130ff8aac51a4b74a3e42e5cd2d72259"
format_stamp: "Formatted at 2026-09-26 02:13:43 on dist-test-slave-hllj"
server_key: "5beecc39e491681071f52d3a14944bb0"
server_key_iv: "a48507d303da536b726a89dc8f510c51"
server_key_version: "encryptionkey@0"
I20260926 02:13:43.064882 32699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:13:43.068446 32714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:43.069900 32699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:13:43.070050 32699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "130ff8aac51a4b74a3e42e5cd2d72259"
format_stamp: "Formatted at 2026-09-26 02:13:43 on dist-test-slave-hllj"
server_key: "5beecc39e491681071f52d3a14944bb0"
server_key_iv: "a48507d303da536b726a89dc8f510c51"
server_key_version: "encryptionkey@0"
I20260926 02:13:43.070212 32699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:43.086863 32699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:43.091157 32699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:43.091427 32699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:43.092372 32699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:43.093796 32699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:43.093871 32699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:43.093967 32699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:43.094008 32699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:43.123680 32699 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:35039
I20260926 02:13:43.123728 358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:35039 every 8 connection(s)
I20260926 02:13:43.124966 32699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:13:43.127249 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 32699
I20260926 02:13:43.127445 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:13:43.127997 29282 external_mini_cluster.cc:1428] Setting key 71c4e613cebb423a5bdf07103ebe619a
Sep 26 02:13:43 dist-test-slave-hllj krb5kdc[32326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388823, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:43.146430 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:43.127624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:43089 (local address 127.28.152.190:44971)
0926 02:13:43.127791 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:43.127796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:43.129549 (+ 1753us) server_negotiation.cc:407] Connection header received
0926 02:13:43.132016 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:43.132020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:43.132085 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:43.132164 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:43.134184 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.134986 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.136579 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.137030 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.137154 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:43.140592 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:43.140598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:43.140602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:43.140637 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:43.143696 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:43.144393 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:43.144396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:43.144399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:43.144516 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:43.144884 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:43.144886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:43.144889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:43.145167 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:43.145281 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:43.146007 (+ 726us) server_negotiation.cc:299] Negotiation successful
0926 02:13:43.146174 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 02:13:43.147485 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:43.128331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:44971 (local address 127.28.152.131:43089)
0926 02:13:43.129387 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:43.129441 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:43.131707 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:43.132472 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:43.132484 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:43.133188 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:43.133999 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:43.134024 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.135131 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.135135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:43.136425 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:43.136440 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.137265 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.137271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:43.137489 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:43.138125 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:43.138148 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:43.140363 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:43.143840 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:43.143849 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:43.143865 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:43.144298 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:43.144590 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:43.144593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:43.144595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:43.144776 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:43.145266 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:43.145272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:43.145620 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 02:13:43.145897 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 02:13:43.146322 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":493,"threads_started":1}
I20260926 02:13:43.148998 359 heartbeater.cc:347] Connected to a master server at 127.28.152.190:44971
I20260926 02:13:43.149384 359 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:43.150169 359 heartbeater.cc:511] Master 127.28.152.190:44971 requested a full tablet report, sending...
I20260926 02:13:43.151746 32371 ts_manager.cc:194] Registered new tserver with Master: 130ff8aac51a4b74a3e42e5cd2d72259 (127.28.152.131:35039)
I20260926 02:13:43.152555 32371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:43089
I20260926 02:13:43.156158 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:13:43.163480 32560 heartbeater.cc:503] Master 127.28.152.190:44971 was elected leader, sending a full tablet report...
Sep 26 02:13:43 dist-test-slave-hllj krb5kdc[32326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388823, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 02:13:43 dist-test-slave-hllj krb5kdc[32326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388823, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:43.204629 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:43.189976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.28.152.190:44971)
0926 02:13:43.190195 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:43.190203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:43.190607 (+ 404us) server_negotiation.cc:407] Connection header received
0926 02:13:43.190742 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:43.190745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:43.190812 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:43.190903 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:43.192278 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.193177 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.194742 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.195270 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.195390 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:43.199407 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:43.199413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:43.199416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:43.199457 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:43.201989 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:43.202687 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:43.202692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:43.202695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:43.202833 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:43.203243 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:43.203247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:43.203249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:43.203576 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:43.203801 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:43.204091 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 02:13:43.204305 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 02:13:43.205261 32371 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53632
I20260926 02:13:43.207077 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 32432
Sep 26 02:13:43 dist-test-slave-hllj krb5kdc[32326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388823, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:13:43.222013 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:43.208036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.28.152.129:45671)
0926 02:13:43.208616 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:43.208622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:43.208648 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:13:43.208703 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:43.208709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:43.208931 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:43.209054 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:43.210128 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.211131 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.212338 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.212963 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.213070 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:43.216854 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:43.216869 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:43.216886 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:43.216933 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:43.219558 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:43.220224 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:43.220232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:43.220234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:43.220352 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:43.220625 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:43.220628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:43.220630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:43.220962 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:43.221227 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:43.221449 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 02:13:43.221647 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":284,"threads_started":1}
W20260926 02:13:43.223263 32555 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47108
W20260926 02:13:43.224395 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 32432: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:13:43.224563 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 32432
I20260926 02:13:43.236896 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 32566
Sep 26 02:13:43 dist-test-slave-hllj krb5kdc[32326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388823, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:13:43.252183 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:43.237743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.28.152.130:46075)
0926 02:13:43.238284 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:43.238292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:43.238318 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:13:43.238380 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:43.238387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:43.238597 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:43.238730 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:43.239946 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.240955 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.242334 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.242898 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.242995 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:43.246849 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:43.246891 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:43.246907 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:43.246950 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:43.249690 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:43.250294 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:43.250301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:43.250303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:43.250422 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:43.250759 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:43.250762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:43.250764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:43.251120 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:43.251355 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:43.251661 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 02:13:43.251889 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":286,"threads_started":1}
W20260926 02:13:43.253636 32689 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38528
W20260926 02:13:43.254784 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 32566: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:13:43.254953 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 32566
I20260926 02:13:43.266933 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 32699
Sep 26 02:13:43 dist-test-slave-hllj krb5kdc[32326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388823, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:13:43.282542 372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:43.267916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.28.152.131:35039)
0926 02:13:43.268527 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:43.268534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:43.268562 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:13:43.268617 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:43.268623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:43.268831 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:43.268968 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:43.270368 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.271381 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.272534 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:43.273091 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:43.273245 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:43.277421 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:43.277432 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:43.277448 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:43.277483 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:43.280053 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:43.280685 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:43.280692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:43.280694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:43.280808 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:43.281092 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:43.281095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:43.281097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:43.281477 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:43.281696 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:43.281974 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 02:13:43.282179 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":335,"threads_started":1}
W20260926 02:13:43.283808 354 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50116
W20260926 02:13:43.284884 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 32699: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:13:43.285025 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 32699
I20260926 02:13:43.297663 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 32339
I20260926 02:13:43.298766 32401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:43.398038 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 32339
2026-09-26T02:13:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3511 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:13:43 dist-test-slave-hllj krb5kdc[376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:13:43 dist-test-slave-hllj krb5kdc[376](info): set up 2 sockets
Sep 26 02:13:43 dist-test-slave-hllj krb5kdc[376](info): commencing operation
krb5kdc: starting...
W20260926 02:13:44.418677 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:13:44 dist-test-slave-hllj krb5kdc[376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:13:44.616530 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:37481
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:37481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:44.939947 389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:44.940294 389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:44.940424 389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:44.947914 389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:44.948063 389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:44.948130 389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:44.948205 389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:44.948267 389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:13:44.958405 389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:37481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:44.959838 389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:44.961274 389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:44.974066 394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:44.974030 395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:44.973990 397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:44.975055 389 server_base.cc:1034] running on GCE node
I20260926 02:13:44.975912 389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:44.977474 389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:44.978729 389 hybrid_clock.cc:648] HybridClock initialized: now 1790388824978619 us; error 91 us; skew 500 ppm
I20260926 02:13:44.979116 389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:44 dist-test-slave-hllj krb5kdc[376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388824, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:44.982421 389 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:13:44.984114 389 webserver.cc:460] Webserver started at http://127.28.152.190:37707/ using document root <none> and password file <none>
I20260926 02:13:44.984889 389 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:44.985015 389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:44.985373 389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:44.988519 389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "5f57ffe71473433a86a3c688c18d75ab"
format_stamp: "Formatted at 2026-09-26 02:13:44 on dist-test-slave-hllj"
server_key: "53b4a8fa94696c9ead6c65b1b683d830"
server_key_iv: "93e36b3ff7d37ffa2d385281efd049ee"
server_key_version: "encryptionkey@0"
I20260926 02:13:44.989403 389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "5f57ffe71473433a86a3c688c18d75ab"
format_stamp: "Formatted at 2026-09-26 02:13:44 on dist-test-slave-hllj"
server_key: "53b4a8fa94696c9ead6c65b1b683d830"
server_key_iv: "93e36b3ff7d37ffa2d385281efd049ee"
server_key_version: "encryptionkey@0"
I20260926 02:13:44.995009 389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:13:44.998713 404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:45.000236 389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:45.000415 389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "5f57ffe71473433a86a3c688c18d75ab"
format_stamp: "Formatted at 2026-09-26 02:13:44 on dist-test-slave-hllj"
server_key: "53b4a8fa94696c9ead6c65b1b683d830"
server_key_iv: "93e36b3ff7d37ffa2d385281efd049ee"
server_key_version: "encryptionkey@0"
I20260926 02:13:45.000607 389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:45.033694 389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:45.038446 389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:45.038745 389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:45.055909 389 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:37481
I20260926 02:13:45.055927 455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:37481 every 8 connection(s)
I20260926 02:13:45.057435 389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:45.061400 456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:45.065209 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 389
I20260926 02:13:45.065415 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:45.066076 29282 external_mini_cluster.cc:1428] Setting key 799e82d0be4346b487464f9b9ca9f21a
I20260926 02:13:45.073658 456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab: Bootstrap starting.
I20260926 02:13:45.077526 456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:45.078902 456 log.cc:824] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:45.082041 456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab: No bootstrap required, opened a new log
Sep 26 02:13:45 dist-test-slave-hllj krb5kdc[376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:45.091239 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:45.068239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.28.152.190:37481)
0926 02:13:45.068995 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:45.069011 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:45.069065 (+ 54us) server_negotiation.cc:407] Connection header received
0926 02:13:45.071774 (+ 2709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:45.071806 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:45.072349 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:45.072823 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:45.074268 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:45.076362 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:45.077884 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:45.079017 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:45.079208 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:45.083425 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:45.083434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:45.083452 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:45.083492 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:45.086661 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:45.087612 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:45.087623 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:45.087631 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:45.087826 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:45.088257 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:45.088260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:45.088262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:45.088895 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:45.089186 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:45.089665 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 02:13:45.090125 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":296,"threads_started":1}
I20260926 02:13:45.091204 456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f57ffe71473433a86a3c688c18d75ab" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 37481 } }
I20260926 02:13:45.091676 456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:45.091786 456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f57ffe71473433a86a3c688c18d75ab, State: Initialized, Role: FOLLOWER
I20260926 02:13:45.092509 456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f57ffe71473433a86a3c688c18d75ab" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 37481 } }
I20260926 02:13:45.092792 456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:45.092935 456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:45.093101 456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:45.095531 456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f57ffe71473433a86a3c688c18d75ab" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 37481 } }
I20260926 02:13:45.096030 456 leader_election.cc:302] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f57ffe71473433a86a3c688c18d75ab; no voters:
I20260926 02:13:45.096701 456 leader_election.cc:288] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:45.096818 461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:45.097482 461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [term 1 LEADER]: Becoming Leader. State: Replica: 5f57ffe71473433a86a3c688c18d75ab, State: Running, Role: LEADER
I20260926 02:13:45.098073 461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f57ffe71473433a86a3c688c18d75ab" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 37481 } }
I20260926 02:13:45.098986 456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:45.101084 463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f57ffe71473433a86a3c688c18d75ab. Latest consensus state: current_term: 1 leader_uuid: "5f57ffe71473433a86a3c688c18d75ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f57ffe71473433a86a3c688c18d75ab" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 37481 } } }
I20260926 02:13:45.101485 462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f57ffe71473433a86a3c688c18d75ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f57ffe71473433a86a3c688c18d75ab" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 37481 } } }
I20260926 02:13:45.101744 462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:45.101744 463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:45.104974 469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:45.110019 469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:45.120813 469 catalog_manager.cc:1347] Generated new cluster ID: 08478f17b44346deb170e345d521ab50
I20260926 02:13:45.120954 469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:45.132902 469 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:45.134387 469 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:45.154006 469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f57ffe71473433a86a3c688c18d75ab: Generated new TSK 0
I20260926 02:13:45.155164 469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:13:45.214754 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:37481
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:45.543223 482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:45.543509 482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:45.543602 482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:45.551347 482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:45.551474 482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:45.551592 482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:13:45.562276 482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:37481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:45.563562 482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:45.564846 482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:45.578303 488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:45.578332 490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:45.578353 487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:45.579110 482 server_base.cc:1034] running on GCE node
I20260926 02:13:45.579757 482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:45.580758 482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:45.581992 482 hybrid_clock.cc:648] HybridClock initialized: now 1790388825581922 us; error 54 us; skew 500 ppm
I20260926 02:13:45.582341 482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:45 dist-test-slave-hllj krb5kdc[376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388825, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:45.585506 482 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:13:45.587002 482 webserver.cc:460] Webserver started at http://127.28.152.129:36263/ using document root <none> and password file <none>
I20260926 02:13:45.587715 482 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:45.587832 482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:45.588136 482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:45.591174 482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "f41559663b074c87802a0e3f6e46343e"
format_stamp: "Formatted at 2026-09-26 02:13:45 on dist-test-slave-hllj"
server_key: "0cd2f976bc7b84179c1a4b6bbd5c2f3a"
server_key_iv: "4ca1045f55966c61d76d2fc6af6ee21f"
server_key_version: "encryptionkey@0"
I20260926 02:13:45.591955 482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "f41559663b074c87802a0e3f6e46343e"
format_stamp: "Formatted at 2026-09-26 02:13:45 on dist-test-slave-hllj"
server_key: "0cd2f976bc7b84179c1a4b6bbd5c2f3a"
server_key_iv: "4ca1045f55966c61d76d2fc6af6ee21f"
server_key_version: "encryptionkey@0"
I20260926 02:13:45.597254 482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:13:45.601003 497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:45.602443 482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:13:45.602595 482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "f41559663b074c87802a0e3f6e46343e"
format_stamp: "Formatted at 2026-09-26 02:13:45 on dist-test-slave-hllj"
server_key: "0cd2f976bc7b84179c1a4b6bbd5c2f3a"
server_key_iv: "4ca1045f55966c61d76d2fc6af6ee21f"
server_key_version: "encryptionkey@0"
I20260926 02:13:45.602782 482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:45.629004 482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:45.632997 482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:45.633273 482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:45.634289 482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:45.635650 482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:45.635721 482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:45.635802 482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:45.635843 482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:45.664974 482 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:43415
I20260926 02:13:45.665014 609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:43415 every 8 connection(s)
I20260926 02:13:45.666404 482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:13:45.673629 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 482
I20260926 02:13:45.673799 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:13:45.674420 29282 external_mini_cluster.cc:1428] Setting key 26f8d35c9651ae3db630614197760510
Sep 26 02:13:45 dist-test-slave-hllj krb5kdc[376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388825, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:45.685946 612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:45.668923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:39871 (local address 127.28.152.190:37481)
0926 02:13:45.669349 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:45.669354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:45.670318 (+ 964us) server_negotiation.cc:407] Connection header received
0926 02:13:45.672157 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:45.672162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:45.672246 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:45.672341 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:45.673919 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:45.674675 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:45.676190 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:45.676591 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:45.676700 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:45.680302 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:45.680307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:45.680311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:45.680343 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:45.682774 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:45.683675 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:45.683679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:45.683681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:45.683787 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:45.684211 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:45.684214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:45.684216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:45.684468 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:45.684570 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:45.685507 (+ 937us) server_negotiation.cc:299] Negotiation successful
0926 02:13:45.685677 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":234,"threads_started":1}
I20260926 02:13:45.687489 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:45.669450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:37481 (local address 127.28.152.129:39871)
0926 02:13:45.670149 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:45.670193 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:45.671872 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:45.672596 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:45.672605 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:45.673153 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:45.673748 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:45.673769 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:45.674811 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:45.674815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:45.676054 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:45.676066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:45.676808 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:45.676815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:45.677048 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:45.677715 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:45.677735 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:45.680061 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:45.682931 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:45.682941 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:45.682960 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:45.683565 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:45.683873 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:45.683877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:45.683881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:45.684118 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:45.684582 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:45.684591 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:45.685053 (+ 462us) client_negotiation.cc:769] Sending connection context
0926 02:13:45.685403 (+ 350us) client_negotiation.cc:240] Negotiation successful
0926 02:13:45.685966 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":254,"threads_started":1}
I20260926 02:13:45.689147 610 heartbeater.cc:347] Connected to a master server at 127.28.152.190:37481
I20260926 02:13:45.689464 610 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:45.690184 610 heartbeater.cc:511] Master 127.28.152.190:37481 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
I20260926 02:13:45.692317 421 ts_manager.cc:194] Registered new tserver with Master: f41559663b074c87802a0e3f6e46343e (127.28.152.129:43415)
I20260926 02:13:45.693864 421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:39871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:13:45.718667 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:37481
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:46.043308 616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:46.043637 616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:46.043732 616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:46.051265 616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:46.051397 616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:46.051512 616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:13:46.061551 616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:37481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:46.062805 616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:46.064111 616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:46.076813 621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:46.076865 624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:46.077636 616 server_base.cc:1034] running on GCE node
W20260926 02:13:46.077649 622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:46.078327 616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:46.079324 616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:46.080550 616 hybrid_clock.cc:648] HybridClock initialized: now 1790388826080491 us; error 52 us; skew 500 ppm
I20260926 02:13:46.080876 616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:46 dist-test-slave-hllj krb5kdc[376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388826, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:46.083995 616 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:13:46.085531 616 webserver.cc:460] Webserver started at http://127.28.152.130:37645/ using document root <none> and password file <none>
I20260926 02:13:46.086230 616 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:46.086345 616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:46.086645 616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:46.089679 616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "f838285a6ba84a359ec42062ce104c50"
format_stamp: "Formatted at 2026-09-26 02:13:46 on dist-test-slave-hllj"
server_key: "697abef469b1460dff41ea71dc15b838"
server_key_iv: "e568d529c0b4a5ae36b0819f46231b20"
server_key_version: "encryptionkey@0"
I20260926 02:13:46.090448 616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "f838285a6ba84a359ec42062ce104c50"
format_stamp: "Formatted at 2026-09-26 02:13:46 on dist-test-slave-hllj"
server_key: "697abef469b1460dff41ea71dc15b838"
server_key_iv: "e568d529c0b4a5ae36b0819f46231b20"
server_key_version: "encryptionkey@0"
I20260926 02:13:46.095912 616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:13:46.099717 631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:46.101290 616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:13:46.101465 616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "f838285a6ba84a359ec42062ce104c50"
format_stamp: "Formatted at 2026-09-26 02:13:46 on dist-test-slave-hllj"
server_key: "697abef469b1460dff41ea71dc15b838"
server_key_iv: "e568d529c0b4a5ae36b0819f46231b20"
server_key_version: "encryptionkey@0"
I20260926 02:13:46.101679 616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:46.120645 616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:46.124811 616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:46.125051 616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:46.126034 616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:46.127424 616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:46.127496 616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:46.127611 616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:46.127663 616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:46.156268 616 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:42055
I20260926 02:13:46.156287 743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:42055 every 8 connection(s)
I20260926 02:13:46.157639 616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:13:46.167227 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 616
I20260926 02:13:46.167423 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:13:46.168028 29282 external_mini_cluster.cc:1428] Setting key 435094de439b6c27d56bc05bf63f9212
Sep 26 02:13:46 dist-test-slave-hllj krb5kdc[376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388826, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:46.179325 612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:46.160265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:48219 (local address 127.28.152.190:37481)
0926 02:13:46.160476 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:46.160482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:46.161700 (+ 1218us) server_negotiation.cc:407] Connection header received
0926 02:13:46.163511 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:46.163516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:46.163593 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:46.163687 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:46.165422 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.166179 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.168413 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.168999 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.169107 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:46.173447 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:46.173453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:46.173457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:46.173495 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:46.176038 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.176834 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.176839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.176841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.176978 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.177526 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.177530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.177532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.177826 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:46.177970 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:46.178735 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 02:13:46.178975 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:13:46.180401 746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:46.160793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:37481 (local address 127.28.152.130:48219)
0926 02:13:46.161533 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:46.161577 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:46.163207 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:46.163952 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:46.163961 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:46.164516 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:46.165180 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:46.165211 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.166348 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.166353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:46.168238 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:46.168253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.169242 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.169248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:46.169476 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:46.170253 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:46.170293 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:46.173117 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:46.176202 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:46.176214 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:46.176235 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:46.176691 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:46.177094 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:46.177100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:46.177104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:46.177397 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:46.177963 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:46.177971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:46.178355 (+ 384us) client_negotiation.cc:769] Sending connection context
0926 02:13:46.178618 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 02:13:46.179208 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":307,"threads_started":1}
I20260926 02:13:46.182128 744 heartbeater.cc:347] Connected to a master server at 127.28.152.190:37481
I20260926 02:13:46.182471 744 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:46.183225 744 heartbeater.cc:511] Master 127.28.152.190:37481 requested a full tablet report, sending...
I20260926 02:13:46.184881 421 ts_manager.cc:194] Registered new tserver with Master: f838285a6ba84a359ec42062ce104c50 (127.28.152.130:42055)
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:13:46.185948 421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:48219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:13:46.213112 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:37481
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:46.537871 749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:46.538195 749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:46.538326 749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:46.545851 749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:46.545992 749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:46.546144 749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:13:46.556144 749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:37481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:46.557600 749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:46.559053 749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:46.573050 757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:46.573107 754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:46.573179 755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:46.573474 749 server_base.cc:1034] running on GCE node
I20260926 02:13:46.574463 749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:46.575430 749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:46.576632 749 hybrid_clock.cc:648] HybridClock initialized: now 1790388826576569 us; error 60 us; skew 500 ppm
I20260926 02:13:46.576958 749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:46 dist-test-slave-hllj krb5kdc[376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388826, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:46.587196 749 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:13:46.588869 749 webserver.cc:460] Webserver started at http://127.28.152.131:39569/ using document root <none> and password file <none>
I20260926 02:13:46.589681 749 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:46.589811 749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:46.590121 749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:46.593290 749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "e3e0eb5548384ae9a4097d104173f369"
format_stamp: "Formatted at 2026-09-26 02:13:46 on dist-test-slave-hllj"
server_key: "acdf8e7969a41cb6d30c45a42c32df95"
server_key_iv: "db36e291c55ae3168faa2ec6dc6d539b"
server_key_version: "encryptionkey@0"
I20260926 02:13:46.594154 749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "e3e0eb5548384ae9a4097d104173f369"
format_stamp: "Formatted at 2026-09-26 02:13:46 on dist-test-slave-hllj"
server_key: "acdf8e7969a41cb6d30c45a42c32df95"
server_key_iv: "db36e291c55ae3168faa2ec6dc6d539b"
server_key_version: "encryptionkey@0"
I20260926 02:13:46.600194 749 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 02:13:46.604218 764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:46.606000 749 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:13:46.606207 749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "e3e0eb5548384ae9a4097d104173f369"
format_stamp: "Formatted at 2026-09-26 02:13:46 on dist-test-slave-hllj"
server_key: "acdf8e7969a41cb6d30c45a42c32df95"
server_key_iv: "db36e291c55ae3168faa2ec6dc6d539b"
server_key_version: "encryptionkey@0"
I20260926 02:13:46.606403 749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:46.637638 749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:46.641877 749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:46.642171 749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:46.643141 749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:46.644511 749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:46.644586 749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:46.644676 749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:46.644726 749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:46.676051 749 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:41271
I20260926 02:13:46.676090 876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:41271 every 8 connection(s)
I20260926 02:13:46.677414 749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:13:46.682523 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 749
I20260926 02:13:46.682696 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:13:46.683233 29282 external_mini_cluster.cc:1428] Setting key 86f5a453438e369cf9266f8e0618f5bf
Sep 26 02:13:46 dist-test-slave-hllj krb5kdc[376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388826, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:46.697587 610 heartbeater.cc:503] Master 127.28.152.190:37481 was elected leader, sending a full tablet report...
I20260926 02:13:46.698140 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:46.679844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:54253 (local address 127.28.152.190:37481)
0926 02:13:46.680281 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:46.680287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:46.681334 (+ 1047us) server_negotiation.cc:407] Connection header received
0926 02:13:46.683855 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:46.683861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:46.683966 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:46.684083 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:46.685979 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.686844 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.688465 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.688869 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.688995 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:46.692477 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:46.692482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:46.692486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:46.692519 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:46.694843 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.695578 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.695582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.695585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.695704 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.696154 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.696157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.696159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.696417 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:46.696524 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:46.697541 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0926 02:13:46.697766 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":243,"threads_started":1}
I20260926 02:13:46.699424 880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:46.680361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:37481 (local address 127.28.152.131:54253)
0926 02:13:46.681091 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:46.681180 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:46.683421 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:46.684383 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:46.684401 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:46.685013 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:46.685780 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:46.685800 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.687001 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.687004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:46.688316 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:46.688331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.689080 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.689084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:46.689365 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:46.690073 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:46.690097 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:46.692234 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:46.694996 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:46.695005 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:46.695024 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:46.695460 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:46.695780 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:46.695783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:46.695786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:46.696052 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:46.696587 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:46.696594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:46.697063 (+ 469us) client_negotiation.cc:769] Sending connection context
0926 02:13:46.697424 (+ 361us) client_negotiation.cc:240] Negotiation successful
0926 02:13:46.697989 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":279,"threads_started":1}
I20260926 02:13:46.700781 877 heartbeater.cc:347] Connected to a master server at 127.28.152.190:37481
I20260926 02:13:46.701148 877 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:46.701908 877 heartbeater.cc:511] Master 127.28.152.190:37481 requested a full tablet report, sending...
I20260926 02:13:46.703329 421 ts_manager.cc:194] Registered new tserver with Master: e3e0eb5548384ae9a4097d104173f369 (127.28.152.131:41271)
I20260926 02:13:46.704167 421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:54253
I20260926 02:13:46.711293 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:13:46 dist-test-slave-hllj krb5kdc[376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:13:46 dist-test-slave-hllj krb5kdc[376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:13:46.757733 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:46.742753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.28.152.129:43415)
0926 02:13:46.743386 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:46.743392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:46.743428 (+ 36us) server_negotiation.cc:407] Connection header received
0926 02:13:46.743490 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:46.743497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:46.743706 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:46.743864 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:46.745266 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.746332 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.747697 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.748332 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.748448 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:46.752279 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:46.752294 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:46.752311 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:46.752352 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:46.755064 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.755769 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.755778 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.755780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.755913 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.756224 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.756227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.756229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.756592 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:46.756823 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:46.757075 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 02:13:46.757337 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":330,"threads_started":1}
Sep 26 02:13:46 dist-test-slave-hllj krb5kdc[376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:46.776005 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:46.762199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.28.152.190:37481)
0926 02:13:46.762421 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:46.762427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:46.762553 (+ 126us) server_negotiation.cc:407] Connection header received
0926 02:13:46.762701 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:46.762705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:46.762768 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:46.762838 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:46.764137 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.764856 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.766237 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.766761 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.766883 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:46.770849 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:46.770854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:46.770857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:46.770898 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:46.773487 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.774218 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.774222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.774224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.774346 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.774712 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.774715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.774717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.775070 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:46.775247 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:46.775527 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 02:13:46.775709 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 02:13:46.776547 421 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34140
I20260926 02:13:46.777595 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 482
I20260926 02:13:46.791004 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:46.778484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.28.152.129:43415)
0926 02:13:46.778699 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:46.778706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:46.778945 (+ 239us) server_negotiation.cc:407] Connection header received
0926 02:13:46.779125 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:46.779129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:46.779223 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:46.779310 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:46.780609 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.781380 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.782652 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.783092 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.783202 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:46.785739 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:46.785745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:46.785748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:46.785785 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:46.788353 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.789098 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.789104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.789107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.789281 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.789653 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.789658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.789662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.789928 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:46.790145 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:46.790465 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 02:13:46.790691 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:13:46.791595 605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:46.873868 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 482
I20260926 02:13:46.886914 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 616
Sep 26 02:13:46 dist-test-slave-hllj krb5kdc[376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:13:46.903820 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:46.887853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.28.152.130:42055)
0926 02:13:46.888437 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:46.888445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:46.888472 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:13:46.888541 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:46.888548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:46.888762 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:46.888899 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:46.890176 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.891247 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.892705 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:46.893400 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:46.893508 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:46.897854 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:46.897871 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:46.897898 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:46.897943 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:46.900932 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.901710 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.901718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.901720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.901862 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:46.902230 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:46.902235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:46.902237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:46.902619 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:46.902911 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:46.903222 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 02:13:46.903459 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":293,"threads_started":1}
I20260926 02:13:46.905279 739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:46.986610 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 616
I20260926 02:13:46.999554 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 749
Sep 26 02:13:47 dist-test-slave-hllj krb5kdc[376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:13:47.016232 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:47.000543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.28.152.131:41271)
0926 02:13:47.001253 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:47.001260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:47.001290 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:13:47.001352 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:47.001360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:47.001578 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:47.001709 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:47.002954 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:47.004051 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:47.005616 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:47.006233 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:47.006349 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:47.010752 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:47.010768 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:47.010785 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:47.010825 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:47.013536 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:47.014316 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:47.014324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:47.014326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:47.014490 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:47.014803 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:47.014807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:47.014809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:47.015176 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:47.015411 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:47.015679 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 02:13:47.015879 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":383,"threads_started":1}
I20260926 02:13:47.017679 872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:47.099133 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 749
I20260926 02:13:47.112076 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 389
I20260926 02:13:47.113346 451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:47.213171 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 389
2026-09-26T02:13:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3815 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:13:47 dist-test-slave-hllj krb5kdc[897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:13:47 dist-test-slave-hllj krb5kdc[897](info): set up 2 sockets
Sep 26 02:13:47 dist-test-slave-hllj krb5kdc[897](info): commencing operation
krb5kdc: starting...
W20260926 02:13:48.215008 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:13:48 dist-test-slave-hllj krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:13:48.413002 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34341
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:48.736554 910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:48.736863 910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:48.736990 910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:48.744488 910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:48.744622 910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:48.744674 910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:48.744751 910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:48.744817 910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:13:48.757630 910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:48.758929 910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:48.760272 910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:48.771432 915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:48.771364 916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:48.771401 918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:48.772450 910 server_base.cc:1034] running on GCE node
I20260926 02:13:48.773280 910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:48.774735 910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:48.775957 910 hybrid_clock.cc:648] HybridClock initialized: now 1790388828775877 us; error 69 us; skew 500 ppm
I20260926 02:13:48.776316 910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:48 dist-test-slave-hllj krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388828, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:48.779528 910 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:13:48.780807 910 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:48.780908 910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:48.781286 910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:48.784340 910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "35559559b8c54fcea6d5409f2a660a99"
format_stamp: "Formatted at 2026-09-26 02:13:48 on dist-test-slave-hllj"
server_key: "f3cfaaed2cb6969bc96ebbda923b8ede"
server_key_iv: "fba2e7cd71fe54bbd67da7f925d71552"
server_key_version: "encryptionkey@0"
I20260926 02:13:48.785207 910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "35559559b8c54fcea6d5409f2a660a99"
format_stamp: "Formatted at 2026-09-26 02:13:48 on dist-test-slave-hllj"
server_key: "f3cfaaed2cb6969bc96ebbda923b8ede"
server_key_iv: "fba2e7cd71fe54bbd67da7f925d71552"
server_key_version: "encryptionkey@0"
I20260926 02:13:48.790486 910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 02:13:48.794245 924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:48.795791 910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:48.796011 910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "35559559b8c54fcea6d5409f2a660a99"
format_stamp: "Formatted at 2026-09-26 02:13:48 on dist-test-slave-hllj"
server_key: "f3cfaaed2cb6969bc96ebbda923b8ede"
server_key_iv: "fba2e7cd71fe54bbd67da7f925d71552"
server_key_version: "encryptionkey@0"
I20260926 02:13:48.796219 910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:48.815186 910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:48.819481 910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:48.819728 910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:48.837234 910 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:34341
I20260926 02:13:48.837257 975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:34341 every 8 connection(s)
I20260926 02:13:48.838147 910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:48.841917 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 910
I20260926 02:13:48.842092 976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:48.842133 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:48.842967 29282 external_mini_cluster.cc:1428] Setting key d9e580c7069cbcb1e34491f0b811a4f4
I20260926 02:13:48.856120 976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99: Bootstrap starting.
Sep 26 02:13:48 dist-test-slave-hllj krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:48.860296 976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:48.861572 976 log.cc:824] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:48.865008 976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99: No bootstrap required, opened a new log
I20260926 02:13:48.868103 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:48.845077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.28.152.190:34341)
0926 02:13:48.845912 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:48.845925 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:48.845968 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:13:48.848319 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:48.848346 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:48.848800 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:48.849231 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:48.851073 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:48.852453 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:48.854347 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:48.855085 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:48.855226 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:48.860102 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:48.860112 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:48.860128 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:48.860173 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:48.863612 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:48.864596 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:48.864604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:48.864613 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:48.864783 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:48.865246 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:48.865250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:48.865252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:48.865887 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:48.866123 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:48.866569 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 02:13:48.866987 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":336,"threads_started":1}
I20260926 02:13:48.873620 976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35559559b8c54fcea6d5409f2a660a99" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:13:48.873984 976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:48.874081 976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35559559b8c54fcea6d5409f2a660a99, State: Initialized, Role: FOLLOWER
I20260926 02:13:48.874660 976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35559559b8c54fcea6d5409f2a660a99" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:13:48.874884 976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:48.875021 976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:48.875200 976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:48.877403 976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35559559b8c54fcea6d5409f2a660a99" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:13:48.877909 976 leader_election.cc:302] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35559559b8c54fcea6d5409f2a660a99; no voters:
I20260926 02:13:48.878474 976 leader_election.cc:288] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:48.878610 981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:48.879226 981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [term 1 LEADER]: Becoming Leader. State: Replica: 35559559b8c54fcea6d5409f2a660a99, State: Running, Role: LEADER
I20260926 02:13:48.879709 981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35559559b8c54fcea6d5409f2a660a99" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:13:48.880452 976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:48.882381 982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35559559b8c54fcea6d5409f2a660a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35559559b8c54fcea6d5409f2a660a99" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } } }
I20260926 02:13:48.882747 982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:48.883270 983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35559559b8c54fcea6d5409f2a660a99. Latest consensus state: current_term: 1 leader_uuid: "35559559b8c54fcea6d5409f2a660a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35559559b8c54fcea6d5409f2a660a99" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } } }
I20260926 02:13:48.883610 983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:48.884675 987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:48.890354 987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:48.899287 987 catalog_manager.cc:1347] Generated new cluster ID: 43fee3ff7fb14a279b2cda13b0bb8eff
I20260926 02:13:48.899443 987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:48.908818 987 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:48.910816 987 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:48.922905 987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99: Generated new TSK 0
I20260926 02:13:48.923913 987 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:13:48.980561 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:49.322048 1002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:49.322346 1002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:49.322494 1002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:49.330188 1002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:49.330312 1002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:49.330473 1002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:13:49.340401 1002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:49.341804 1002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:49.343087 1002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:49.355727 1008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:49.355865 1007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:49.356268 1010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:49.356504 1002 server_base.cc:1034] running on GCE node
I20260926 02:13:49.357122 1002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:49.358139 1002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:49.359347 1002 hybrid_clock.cc:648] HybridClock initialized: now 1790388829359293 us; error 39 us; skew 500 ppm
I20260926 02:13:49.359681 1002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:49 dist-test-slave-hllj krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388829, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:49.362833 1002 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:13:49.363981 1002 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:49.364087 1002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:49.364382 1002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:49.367542 1002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "e1b85ea495854b5da103c169522706b1"
format_stamp: "Formatted at 2026-09-26 02:13:49 on dist-test-slave-hllj"
server_key: "58ded5f756546f592cfabe717c2aced0"
server_key_iv: "ffa5616f46808fbb2128125e3d275f1f"
server_key_version: "encryptionkey@0"
I20260926 02:13:49.368330 1002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "e1b85ea495854b5da103c169522706b1"
format_stamp: "Formatted at 2026-09-26 02:13:49 on dist-test-slave-hllj"
server_key: "58ded5f756546f592cfabe717c2aced0"
server_key_iv: "ffa5616f46808fbb2128125e3d275f1f"
server_key_version: "encryptionkey@0"
I20260926 02:13:49.373451 1002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:13:49.376989 1016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:49.378460 1002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:13:49.378618 1002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "e1b85ea495854b5da103c169522706b1"
format_stamp: "Formatted at 2026-09-26 02:13:49 on dist-test-slave-hllj"
server_key: "58ded5f756546f592cfabe717c2aced0"
server_key_iv: "ffa5616f46808fbb2128125e3d275f1f"
server_key_version: "encryptionkey@0"
I20260926 02:13:49.378821 1002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:49.400543 1002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:49.404982 1002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:49.405303 1002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:49.406359 1002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:49.407823 1002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:49.407944 1002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:49.408047 1002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:49.408116 1002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:49.438177 1002 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:36055
I20260926 02:13:49.438228 1128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:36055 every 8 connection(s)
I20260926 02:13:49.439047 1002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:13:49.439556 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 1002
I20260926 02:13:49.439741 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:13:49.440507 29282 external_mini_cluster.cc:1428] Setting key 72f4ffdd7c7e457306d0945b5600e4fa
Sep 26 02:13:49 dist-test-slave-hllj krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388829, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
I20260926 02:13:49.459995 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:49.441712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:38667 (local address 127.28.152.190:34341)
0926 02:13:49.442150 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:49.442155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:49.443385 (+ 1230us) server_negotiation.cc:407] Connection header received
0926 02:13:49.445305 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:49.445311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:49.445397 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:49.445504 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:49.447197 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:49.447918 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:49.449715 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:49.450118 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:49.450234 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:49.453953 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:49.453959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:49.453962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:49.453997 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:49.456728 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:49.457604 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:49.457609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:49.457611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:49.457734 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:49.458152 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:49.458156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:49.458158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:49.458471 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:49.458601 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:49.459365 (+ 764us) server_negotiation.cc:299] Negotiation successful
0926 02:13:49.459603 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":238,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
I20260926 02:13:49.461117 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:49.442454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:38667)
0926 02:13:49.443227 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:49.443267 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:49.444985 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:49.445792 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:49.445801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:49.446364 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:49.447005 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:49.447026 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:49.448091 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:49.448095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:49.449529 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:49.449544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:49.450369 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:49.450376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:49.450604 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:49.451277 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:49.451300 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:49.453687 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:49.456902 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:49.456912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:49.456932 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:49.457469 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:49.457841 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:49.457845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:49.457848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:49.458036 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:49.458594 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:49.458603 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:49.458976 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 02:13:49.459232 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 02:13:49.459839 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":301,"threads_started":1}
I20260926 02:13:49.462875 1129 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34341
I20260926 02:13:49.463192 1129 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:49.463892 1129 heartbeater.cc:511] Master 127.28.152.190:34341 requested a full tablet report, sending...
I20260926 02:13:49.466153 941 ts_manager.cc:194] Registered new tserver with Master: e1b85ea495854b5da103c169522706b1 (127.28.152.129:36055)
I20260926 02:13:49.467859 941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:38667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:13:49.484952 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:49.811391 1135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:49.811712 1135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:49.811847 1135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:49.819721 1135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:49.819871 1135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:49.820035 1135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:13:49.830686 1135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:49.832170 1135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:49.833611 1135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:49.848544 1140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:49.848496 1141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:49.849236 1135 server_base.cc:1034] running on GCE node
W20260926 02:13:49.849345 1143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:49.849980 1135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:49.851009 1135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:49.852252 1135 hybrid_clock.cc:648] HybridClock initialized: now 1790388829852172 us; error 67 us; skew 500 ppm
I20260926 02:13:49.852632 1135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:49 dist-test-slave-hllj krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388829, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:49.856000 1135 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:13:49.857242 1135 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:49.857371 1135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:49.857684 1135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:49.860723 1135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "f73c3be01d7f403998b2d6fb9867c43e"
format_stamp: "Formatted at 2026-09-26 02:13:49 on dist-test-slave-hllj"
server_key: "1314fd9f8f6b1a018572b82ca6228841"
server_key_iv: "bf52597d8ddb0fa9c2276a6e7d666b4f"
server_key_version: "encryptionkey@0"
I20260926 02:13:49.861632 1135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "f73c3be01d7f403998b2d6fb9867c43e"
format_stamp: "Formatted at 2026-09-26 02:13:49 on dist-test-slave-hllj"
server_key: "1314fd9f8f6b1a018572b82ca6228841"
server_key_iv: "bf52597d8ddb0fa9c2276a6e7d666b4f"
server_key_version: "encryptionkey@0"
I20260926 02:13:49.866999 1135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:13:49.870666 1149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:49.872071 1135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:13:49.872229 1135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "f73c3be01d7f403998b2d6fb9867c43e"
format_stamp: "Formatted at 2026-09-26 02:13:49 on dist-test-slave-hllj"
server_key: "1314fd9f8f6b1a018572b82ca6228841"
server_key_iv: "bf52597d8ddb0fa9c2276a6e7d666b4f"
server_key_version: "encryptionkey@0"
I20260926 02:13:49.872447 1135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:49.885752 1135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:49.890616 1135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:49.890919 1135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:49.891960 1135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:49.893481 1135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:49.893581 1135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:49.893692 1135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:49.893760 1135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:49.927265 1135 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:33515
I20260926 02:13:49.927325 1261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:33515 every 8 connection(s)
I20260926 02:13:49.928133 1135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:13:49.934345 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 1135
I20260926 02:13:49.934535 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:13:49.935021 29282 external_mini_cluster.cc:1428] Setting key 393ed7b5a541302baf5892068c08a26b
Sep 26 02:13:49 dist-test-slave-hllj krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388829, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:49.950448 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:49.930720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:56079 (local address 127.28.152.190:34341)
0926 02:13:49.930943 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:49.930949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:49.932227 (+ 1278us) server_negotiation.cc:407] Connection header received
0926 02:13:49.934190 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:49.934196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:49.934320 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:49.934448 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:49.936195 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:49.936996 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:49.938674 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:49.939290 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:49.939408 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:49.943334 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:49.943341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:49.943346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:49.943385 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:49.946596 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:49.947670 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:49.947675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:49.947678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:49.947844 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:49.948381 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:49.948385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:49.948388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:49.948747 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:49.948897 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:49.949815 (+ 918us) server_negotiation.cc:299] Negotiation successful
0926 02:13:49.950064 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:13:49.951507 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:49.931302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:56079)
0926 02:13:49.932059 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:49.932099 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:49.933842 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:49.934753 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:49.934764 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:49.935383 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:49.936009 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:49.936035 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:49.937185 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:49.937190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:49.938488 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:49.938507 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:49.939513 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:49.939517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:49.939738 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:49.940399 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:49.940426 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:49.943007 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:49.946822 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:49.946830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:49.946847 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:49.947512 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:49.948013 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:49.948019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:49.948023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:49.948253 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:49.948936 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:49.948944 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:49.949412 (+ 468us) client_negotiation.cc:769] Sending connection context
0926 02:13:49.949683 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 02:13:49.950128 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":295,"threads_started":1}
I20260926 02:13:49.953318 1262 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34341
I20260926 02:13:49.953698 1262 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:49.954622 1262 heartbeater.cc:511] Master 127.28.152.190:34341 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:13:49.956457 941 ts_manager.cc:194] Registered new tserver with Master: f73c3be01d7f403998b2d6fb9867c43e (127.28.152.130:33515)
I20260926 02:13:49.957454 941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:56079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:13:49.981470 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:13:50.304332 1267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:50.304620 1267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:50.304741 1267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:50.312211 1267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:50.312343 1267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:50.312489 1267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:13:50.323756 1267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:50.325493 1267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:50.327114 1267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:50.341805 1272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:50.342442 1273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:50.343505 1275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:50.343969 1267 server_base.cc:1034] running on GCE node
I20260926 02:13:50.344583 1267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:50.345649 1267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:50.346895 1267 hybrid_clock.cc:648] HybridClock initialized: now 1790388830346790 us; error 69 us; skew 500 ppm
I20260926 02:13:50.347231 1267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:13:50 dist-test-slave-hllj krb5kdc[897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388830, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:13:50.350414 1267 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:13:50.351670 1267 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:50.351778 1267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:50.352093 1267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:50.355082 1267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "53a6a5e714464066b68e1dcdae5e9c91"
format_stamp: "Formatted at 2026-09-26 02:13:50 on dist-test-slave-hllj"
server_key: "653928e21b6dcd9f2968cca784e744c7"
server_key_iv: "3e38887d1645357dd43464617c703e77"
server_key_version: "encryptionkey@0"
I20260926 02:13:50.355887 1267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "53a6a5e714464066b68e1dcdae5e9c91"
format_stamp: "Formatted at 2026-09-26 02:13:50 on dist-test-slave-hllj"
server_key: "653928e21b6dcd9f2968cca784e744c7"
server_key_iv: "3e38887d1645357dd43464617c703e77"
server_key_version: "encryptionkey@0"
I20260926 02:13:50.361193 1267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:13:50.364816 1281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:50.366307 1267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:13:50.366469 1267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "53a6a5e714464066b68e1dcdae5e9c91"
format_stamp: "Formatted at 2026-09-26 02:13:50 on dist-test-slave-hllj"
server_key: "653928e21b6dcd9f2968cca784e744c7"
server_key_iv: "3e38887d1645357dd43464617c703e77"
server_key_version: "encryptionkey@0"
I20260926 02:13:50.366676 1267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:50.404330 1267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:50.408926 1267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:50.409237 1267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:50.410290 1267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:50.411715 1267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:50.411813 1267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:50.411926 1267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:50.412002 1267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:50.441990 1267 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:46439
I20260926 02:13:50.442018 1393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:46439 every 8 connection(s)
I20260926 02:13:50.442855 1267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:13:50.451066 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 1267
I20260926 02:13:50.451259 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:13:50.451799 29282 external_mini_cluster.cc:1428] Setting key 4f1302c83147e7b50342e68daecd6eed
Sep 26 02:13:50 dist-test-slave-hllj krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388830, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:13:50.463493 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.445367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:40039 (local address 127.28.152.190:34341)
0926 02:13:50.445559 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:50.445565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:50.446812 (+ 1247us) server_negotiation.cc:407] Connection header received
0926 02:13:50.448584 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:50.448588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:50.448695 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:50.448788 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:50.450426 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.451603 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.453567 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.454125 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.454222 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.457814 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:50.457820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:50.457823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:50.457859 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:50.460509 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:50.461332 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:50.461337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:50.461339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:50.461466 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:50.461827 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:50.461830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:50.461832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:50.462168 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:50.462281 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:50.463065 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 02:13:50.463234 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:13:50.464542 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.445908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:40039)
0926 02:13:50.446642 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:50.446691 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:50.448296 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:50.448979 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:50.448988 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:50.449605 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:50.450229 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.450246 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.451746 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.451750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.453409 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.453423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.454362 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.454371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.454687 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.455419 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:13:50.455442 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:50.457589 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:50.460666 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:50.460676 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:50.460691 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:50.461216 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:50.461553 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:13:50.461557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:13:50.461559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:13:50.461739 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:13:50.462296 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:50.462302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:50.462665 (+ 363us) client_negotiation.cc:769] Sending connection context
0926 02:13:50.462971 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 02:13:50.463395 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":285,"threads_started":1}
I20260926 02:13:50.466032 1394 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34341
I20260926 02:13:50.466346 1394 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:50.467105 1394 heartbeater.cc:511] Master 127.28.152.190:34341 requested a full tablet report, sending...
I20260926 02:13:50.468413 941 ts_manager.cc:194] Registered new tserver with Master: 53a6a5e714464066b68e1dcdae5e9c91 (127.28.152.131:46439)
I20260926 02:13:50.469219 941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:40039
I20260926 02:13:50.471066 1129 heartbeater.cc:503] Master 127.28.152.190:34341 was elected leader, sending a full tablet report...
I20260926 02:13:50.479418 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:13:50.496271 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.483777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.28.152.190:34341)
0926 02:13:50.483932 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:50.483936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:50.484439 (+ 503us) server_negotiation.cc:407] Connection header received
0926 02:13:50.484607 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:50.484612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:50.484712 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:50.484796 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:50.486358 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.487065 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.488330 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.488758 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.488864 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.491034 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:50.491039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:50.491042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:50.491075 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:50.493627 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:50.494305 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:50.494311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:50.494315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:50.494450 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:50.494803 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:50.494807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:50.494811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:50.495150 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:50.495375 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:50.495740 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 02:13:50.495955 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:13:50.500329 941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:13:50.507397 941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:13:50.532305 1408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.520260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33515 (local address 127.0.0.1:58322)
0926 02:13:50.522296 (+ 2036us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:50.522358 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:50.522561 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:50.523661 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:50.523669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:50.523708 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:50.524116 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.524128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.526208 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.526215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.528797 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.528814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.531204 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.531211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.531453 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.531508 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 02:13:50.531652 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 02:13:50.531868 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1747,"threads_started":1}
I20260926 02:13:50.533464 1407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.520466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.28.152.130:33515)
0926 02:13:50.521679 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:50.521690 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:50.522771 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 02:13:50.522883 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:50.522892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:50.523195 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:50.523377 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:50.524327 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.526037 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.529014 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.530669 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.530844 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.531060 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:50.532873 (+ 1813us) server_negotiation.cc:299] Negotiation successful
0926 02:13:50.533098 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1028,"thread_start_us":875,"threads_started":1}
I20260926 02:13:50.539633 1197 tablet_service.cc:1467] Processing CreateTablet for tablet 85a60cb7f98b45ec9e011a41ca3e5a35 (DEFAULT_TABLE table=test-table [id=1521493e0c294152a0d4647e614154e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:50.541018 1197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a60cb7f98b45ec9e011a41ca3e5a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:50.545260 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.523088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.28.152.129:36055)
0926 02:13:50.537174 (+ 14086us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:50.537182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:50.537213 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:13:50.537312 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:50.537320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:50.537588 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:50.537784 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:50.538864 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.540343 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.542607 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.544210 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.544396 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.544579 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:50.544784 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 02:13:50.544937 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13904,"thread_start_us":13753,"threads_started":1}
I20260926 02:13:50.545293 1406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.522437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:36055 (local address 127.0.0.1:44944)
0926 02:13:50.535090 (+ 12653us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:50.535105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:50.535193 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:50.538007 (+ 2814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:50.538012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:50.538036 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:50.538372 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.538384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.540452 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.540457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.542472 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.542481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.544417 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.544423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.544606 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.544639 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:13:50.544719 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 02:13:50.544855 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12558,"mutex_wait_us":23,"thread_start_us":16191,"threads_started":1}
I20260926 02:13:50.545751 1408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.520649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:46439 (local address 127.0.0.1:56796)
0926 02:13:50.532571 (+ 11922us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:50.532593 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:50.532701 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:50.538621 (+ 5920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:50.538627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:50.538652 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:50.538879 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.538890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.540205 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.540210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.542216 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.542245 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.544287 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.544293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.544494 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.544528 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:13:50.544618 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 02:13:50.544753 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11814,"mutex_wait_us":43,"thread_start_us":1253,"threads_started":1}
I20260926 02:13:50.546097 1409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.521359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.28.152.131:46439)
0926 02:13:50.536418 (+ 15059us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:50.536428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:50.536471 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:13:50.536574 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:50.536582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:50.536853 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:50.538294 (+ 1441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:50.539058 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.540063 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.542411 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.543839 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.544024 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.544192 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:50.544966 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 02:13:50.545161 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14865,"thread_start_us":15167,"threads_started":1}
I20260926 02:13:50.549755 1329 tablet_service.cc:1467] Processing CreateTablet for tablet 85a60cb7f98b45ec9e011a41ca3e5a35 (DEFAULT_TABLE table=test-table [id=1521493e0c294152a0d4647e614154e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:50.549564 1064 tablet_service.cc:1467] Processing CreateTablet for tablet 85a60cb7f98b45ec9e011a41ca3e5a35 (DEFAULT_TABLE table=test-table [id=1521493e0c294152a0d4647e614154e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:50.551656 1329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a60cb7f98b45ec9e011a41ca3e5a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:50.551915 1064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a60cb7f98b45ec9e011a41ca3e5a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:50.555243 1412 tablet_bootstrap.cc:492] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: Bootstrap starting.
I20260926 02:13:50.559068 1412 tablet_bootstrap.cc:654] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:50.560468 1412 log.cc:824] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:50.563796 1412 tablet_bootstrap.cc:492] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: No bootstrap required, opened a new log
I20260926 02:13:50.564096 1412 ts_tablet_manager.cc:1397] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260926 02:13:50.564299 1414 tablet_bootstrap.cc:492] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1: Bootstrap starting.
I20260926 02:13:50.565399 1415 tablet_bootstrap.cc:492] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91: Bootstrap starting.
I20260926 02:13:50.568027 1414 tablet_bootstrap.cc:654] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:50.568789 1415 tablet_bootstrap.cc:654] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:50.569412 1414 log.cc:824] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:50.570117 1415 log.cc:824] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:50.572487 1412 raft_consensus.cc:348] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73c3be01d7f403998b2d6fb9867c43e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33515 } } peers { permanent_uuid: "53a6a5e714464066b68e1dcdae5e9c91" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46439 } } peers { permanent_uuid: "e1b85ea495854b5da103c169522706b1" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 36055 } }
I20260926 02:13:50.572866 1412 raft_consensus.cc:374] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:50.572969 1412 raft_consensus.cc:729] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73c3be01d7f403998b2d6fb9867c43e, State: Initialized, Role: FOLLOWER
I20260926 02:13:50.573599 1412 consensus_queue.cc:260] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73c3be01d7f403998b2d6fb9867c43e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33515 } } peers { permanent_uuid: "53a6a5e714464066b68e1dcdae5e9c91" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46439 } } peers { permanent_uuid: "e1b85ea495854b5da103c169522706b1" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 36055 } }
I20260926 02:13:50.574649 1262 heartbeater.cc:503] Master 127.28.152.190:34341 was elected leader, sending a full tablet report...
I20260926 02:13:50.575363 1412 ts_tablet_manager.cc:1428] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260926 02:13:50.578671 1414 tablet_bootstrap.cc:492] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1: No bootstrap required, opened a new log
I20260926 02:13:50.578670 1415 tablet_bootstrap.cc:492] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91: No bootstrap required, opened a new log
I20260926 02:13:50.578985 1415 ts_tablet_manager.cc:1397] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260926 02:13:50.578975 1414 ts_tablet_manager.cc:1397] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260926 02:13:50.589848 1414 raft_consensus.cc:348] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73c3be01d7f403998b2d6fb9867c43e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33515 } } peers { permanent_uuid: "53a6a5e714464066b68e1dcdae5e9c91" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46439 } } peers { permanent_uuid: "e1b85ea495854b5da103c169522706b1" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 36055 } }
I20260926 02:13:50.589848 1415 raft_consensus.cc:348] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73c3be01d7f403998b2d6fb9867c43e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33515 } } peers { permanent_uuid: "53a6a5e714464066b68e1dcdae5e9c91" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46439 } } peers { permanent_uuid: "e1b85ea495854b5da103c169522706b1" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 36055 } }
I20260926 02:13:50.590363 1414 raft_consensus.cc:374] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:50.590363 1415 raft_consensus.cc:374] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:50.590469 1415 raft_consensus.cc:729] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a6a5e714464066b68e1dcdae5e9c91, State: Initialized, Role: FOLLOWER
I20260926 02:13:50.590492 1414 raft_consensus.cc:729] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b85ea495854b5da103c169522706b1, State: Initialized, Role: FOLLOWER
I20260926 02:13:50.591184 1414 consensus_queue.cc:260] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73c3be01d7f403998b2d6fb9867c43e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33515 } } peers { permanent_uuid: "53a6a5e714464066b68e1dcdae5e9c91" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46439 } } peers { permanent_uuid: "e1b85ea495854b5da103c169522706b1" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 36055 } }
I20260926 02:13:50.591184 1415 consensus_queue.cc:260] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73c3be01d7f403998b2d6fb9867c43e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33515 } } peers { permanent_uuid: "53a6a5e714464066b68e1dcdae5e9c91" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46439 } } peers { permanent_uuid: "e1b85ea495854b5da103c169522706b1" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 36055 } }
I20260926 02:13:50.592303 1394 heartbeater.cc:503] Master 127.28.152.190:34341 was elected leader, sending a full tablet report...
I20260926 02:13:50.593169 1415 ts_tablet_manager.cc:1428] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 02:13:50.593319 1414 ts_tablet_manager.cc:1428] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260926 02:13:50.597424 1420 raft_consensus.cc:482] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:50.597667 1420 raft_consensus.cc:504] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73c3be01d7f403998b2d6fb9867c43e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33515 } } peers { permanent_uuid: "53a6a5e714464066b68e1dcdae5e9c91" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46439 } } peers { permanent_uuid: "e1b85ea495854b5da103c169522706b1" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 36055 } }
I20260926 02:13:50.599257 1420 leader_election.cc:288] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f73c3be01d7f403998b2d6fb9867c43e (127.28.152.130:33515), e1b85ea495854b5da103c169522706b1 (127.28.152.129:36055)
I20260926 02:13:50.606102 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.599538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:36055 (local address 127.28.152.131:51607)
0926 02:13:50.599750 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:50.599769 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:50.599910 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:50.600665 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:50.600674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:50.600704 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:50.600946 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.600959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.602279 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.602283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.604123 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.604138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.605523 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.605527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.605667 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.605701 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:13:50.605785 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 02:13:50.605884 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":4}
I20260926 02:13:50.606213 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.600147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:51607 (local address 127.28.152.129:36055)
0926 02:13:50.600352 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:50.600359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:50.600376 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:13:50.600446 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:50.600451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:50.600549 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:50.600636 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:50.601097 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.602161 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.604322 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.605369 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.605535 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.605612 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:50.605845 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 02:13:50.605983 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:13:50.607033 1407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.600381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:40563 (local address 127.28.152.130:33515)
0926 02:13:50.600570 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:50.600576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:50.601081 (+ 505us) server_negotiation.cc:407] Connection header received
0926 02:13:50.601203 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:50.601217 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:50.601318 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:50.601416 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:13:50.601843 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.602686 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.604864 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.606041 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.606184 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.606252 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:50.606764 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 02:13:50.606862 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:13:50.607288 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.599653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33515 (local address 127.28.152.131:40563)
0926 02:13:50.600927 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:50.600949 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:50.601070 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:50.601425 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:50.601428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:50.601446 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:13:50.601701 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.601708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.602819 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.602824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.604707 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:50.604718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.606508 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.606513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:50.606624 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.606646 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 02:13:50.606738 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:13:50.606861 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1154,"thread_start_us":953,"threads_started":1}
I20260926 02:13:50.607177 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a60cb7f98b45ec9e011a41ca3e5a35" candidate_uuid: "53a6a5e714464066b68e1dcdae5e9c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b85ea495854b5da103c169522706b1" is_pre_election: true
I20260926 02:13:50.607791 1084 raft_consensus.cc:2454] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53a6a5e714464066b68e1dcdae5e9c91 in term 0.
I20260926 02:13:50.608340 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a60cb7f98b45ec9e011a41ca3e5a35" candidate_uuid: "53a6a5e714464066b68e1dcdae5e9c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f73c3be01d7f403998b2d6fb9867c43e" is_pre_election: true
I20260926 02:13:50.608438 1284 leader_election.cc:302] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a6a5e714464066b68e1dcdae5e9c91, e1b85ea495854b5da103c169522706b1; no voters:
I20260926 02:13:50.608812 1420 raft_consensus.cc:2781] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:50.608815 1217 raft_consensus.cc:2454] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53a6a5e714464066b68e1dcdae5e9c91 in term 0.
I20260926 02:13:50.608973 1420 raft_consensus.cc:482] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:50.609079 1420 raft_consensus.cc:3037] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:50.611368 1420 raft_consensus.cc:504] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73c3be01d7f403998b2d6fb9867c43e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33515 } } peers { permanent_uuid: "53a6a5e714464066b68e1dcdae5e9c91" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46439 } } peers { permanent_uuid: "e1b85ea495854b5da103c169522706b1" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 36055 } }
I20260926 02:13:50.611963 1420 leader_election.cc:288] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [CANDIDATE]: Term 1 election: Requested vote from peers f73c3be01d7f403998b2d6fb9867c43e (127.28.152.130:33515), e1b85ea495854b5da103c169522706b1 (127.28.152.129:36055)
I20260926 02:13:50.612388 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a60cb7f98b45ec9e011a41ca3e5a35" candidate_uuid: "53a6a5e714464066b68e1dcdae5e9c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f73c3be01d7f403998b2d6fb9867c43e"
I20260926 02:13:50.612445 1084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a60cb7f98b45ec9e011a41ca3e5a35" candidate_uuid: "53a6a5e714464066b68e1dcdae5e9c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1b85ea495854b5da103c169522706b1"
I20260926 02:13:50.612604 1217 raft_consensus.cc:3037] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:50.612617 1084 raft_consensus.cc:3037] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:50.615787 1217 raft_consensus.cc:2454] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53a6a5e714464066b68e1dcdae5e9c91 in term 1.
I20260926 02:13:50.615787 1084 raft_consensus.cc:2454] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53a6a5e714464066b68e1dcdae5e9c91 in term 1.
I20260926 02:13:50.616251 1282 leader_election.cc:302] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a6a5e714464066b68e1dcdae5e9c91, f73c3be01d7f403998b2d6fb9867c43e; no voters:
I20260926 02:13:50.616624 1420 raft_consensus.cc:2781] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:50.617043 1420 raft_consensus.cc:686] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 1 LEADER]: Becoming Leader. State: Replica: 53a6a5e714464066b68e1dcdae5e9c91, State: Running, Role: LEADER
I20260926 02:13:50.617583 1420 consensus_queue.cc:237] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73c3be01d7f403998b2d6fb9867c43e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33515 } } peers { permanent_uuid: "53a6a5e714464066b68e1dcdae5e9c91" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46439 } } peers { permanent_uuid: "e1b85ea495854b5da103c169522706b1" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 36055 } }
I20260926 02:13:50.622196 941 catalog_manager.cc:5495] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53a6a5e714464066b68e1dcdae5e9c91 (127.28.152.131). New cstate: current_term: 1 leader_uuid: "53a6a5e714464066b68e1dcdae5e9c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73c3be01d7f403998b2d6fb9867c43e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53a6a5e714464066b68e1dcdae5e9c91" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1b85ea495854b5da103c169522706b1" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 36055 } health_report { overall_health: UNKNOWN } } }
I20260926 02:13:50.656044 1409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.650960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.28.152.131:46439)
0926 02:13:50.651155 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:50.651160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:50.651178 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:13:50.651329 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:50.651333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:50.651408 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:50.651539 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:13:50.651822 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.652579 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.653979 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.654368 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.654463 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.654995 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:13:50.655542 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:13:50.655665 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:50.655710 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 02:13:50.655776 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:13:50.664918 1084 raft_consensus.cc:1261] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1 [term 1 FOLLOWER]: Refusing update from remote peer 53a6a5e714464066b68e1dcdae5e9c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:50.664947 1217 raft_consensus.cc:1261] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e [term 1 FOLLOWER]: Refusing update from remote peer 53a6a5e714464066b68e1dcdae5e9c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:50.666265 1420 consensus_queue.cc:1035] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f73c3be01d7f403998b2d6fb9867c43e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:50.666600 1422 consensus_queue.cc:1035] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1b85ea495854b5da103c169522706b1" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 36055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:50.679244 1425 mvcc.cc:205] Tried to move back new op lower bound from 7333432650391683072 to 7333432650213658624. Current Snapshot: MvccSnapshot[applied={T|T < 7333432650391683072}]
W20260926 02:13:50.680327 1263 tablet.cc:2369] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:13:50.696596 940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:50.696974 940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:50.701910 940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35559559b8c54fcea6d5409f2a660a99: Sending DeleteTablet for 3 replicas of tablet 85a60cb7f98b45ec9e011a41ca3e5a35
I20260926 02:13:50.703243 1197 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a60cb7f98b45ec9e011a41ca3e5a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:50 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:58322
I20260926 02:13:50.713382 1064 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a60cb7f98b45ec9e011a41ca3e5a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:50 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:44944
I20260926 02:13:50.726758 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 1002
I20260926 02:13:50.752355 1436 tablet_replica.cc:330] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1: stopping tablet replica
I20260926 02:13:50.752792 1329 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a60cb7f98b45ec9e011a41ca3e5a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:50 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:56796
I20260926 02:13:50.753037 1436 raft_consensus.cc:2229] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:50.753640 1436 raft_consensus.cc:2258] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:50.753764 1438 tablet_replica.cc:330] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91: stopping tablet replica
I20260926 02:13:50.754200 1438 raft_consensus.cc:2229] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:13:50.754861 1438 raft_consensus.cc:2258] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:50.753692 1435 tablet_replica.cc:330] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: stopping tablet replica
I20260926 02:13:50.755654 1435 raft_consensus.cc:2229] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:50.756150 1436 ts_tablet_manager.cc:1905] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:50.756712 1435 raft_consensus.cc:2258] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:50.756742 1438 ts_tablet_manager.cc:1905] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:50.759370 1435 ts_tablet_manager.cc:1905] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:50.763553 1436 ts_tablet_manager.cc:1918] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:13:50.763777 1436 log.cc:1196] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/85a60cb7f98b45ec9e011a41ca3e5a35
I20260926 02:13:50.763741 1438 ts_tablet_manager.cc:1918] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:13:50.763916 1438 log.cc:1196] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/85a60cb7f98b45ec9e011a41ca3e5a35
I20260926 02:13:50.764339 1436 ts_tablet_manager.cc:1939] T 85a60cb7f98b45ec9e011a41ca3e5a35 P e1b85ea495854b5da103c169522706b1: Deleting consensus metadata
I20260926 02:13:50.764423 1438 ts_tablet_manager.cc:1939] T 85a60cb7f98b45ec9e011a41ca3e5a35 P 53a6a5e714464066b68e1dcdae5e9c91: Deleting consensus metadata
I20260926 02:13:50.766253 928 catalog_manager.cc:4841] TS 53a6a5e714464066b68e1dcdae5e9c91 (127.28.152.131:46439): tablet 85a60cb7f98b45ec9e011a41ca3e5a35 (table test-table [id=1521493e0c294152a0d4647e614154e3]) successfully deleted
I20260926 02:13:50.766297 927 catalog_manager.cc:4841] TS e1b85ea495854b5da103c169522706b1 (127.28.152.129:36055): tablet 85a60cb7f98b45ec9e011a41ca3e5a35 (table test-table [id=1521493e0c294152a0d4647e614154e3]) successfully deleted
Sep 26 02:13:50 dist-test-slave-hllj krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:13:50.772317 1435 ts_tablet_manager.cc:1918] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:13:50.772511 1435 log.cc:1196] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/85a60cb7f98b45ec9e011a41ca3e5a35
I20260926 02:13:50.773079 1435 ts_tablet_manager.cc:1939] T 85a60cb7f98b45ec9e011a41ca3e5a35 P f73c3be01d7f403998b2d6fb9867c43e: Deleting consensus metadata
I20260926 02:13:50.774966 925 catalog_manager.cc:4841] TS f73c3be01d7f403998b2d6fb9867c43e (127.28.152.130:33515): tablet 85a60cb7f98b45ec9e011a41ca3e5a35 (table test-table [id=1521493e0c294152a0d4647e614154e3]) successfully deleted
I20260926 02:13:50.777297 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.728764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.28.152.129:36055)
0926 02:13:50.729002 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:50.729009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:50.729037 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:13:50.730038 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:50.730056 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:50.730165 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:50.730295 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:50.757337 (+ 27042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.758468 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.763898 (+ 5430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.764625 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.764747 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.770073 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:50.770091 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:50.770111 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:50.770177 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:50.774027 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:50.775085 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:50.775095 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:50.775098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:50.775271 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:50.775726 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:50.775729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:50.775731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:50.776146 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:50.776413 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:50.776751 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 02:13:50.776942 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:13:50.777994 1124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:50.875674 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 1002
I20260926 02:13:50.889911 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 1135
Sep 26 02:13:50 dist-test-slave-hllj krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:13:50.904912 1407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:50.890672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.28.152.130:33515)
0926 02:13:50.890836 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:50.890843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:50.890917 (+ 74us) server_negotiation.cc:407] Connection header received
0926 02:13:50.891111 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:50.891117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:50.891181 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:50.891264 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:50.892458 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.893247 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.894491 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:50.894876 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:50.894988 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:50.899050 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:50.899067 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:50.899084 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:50.899131 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:50.902065 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:50.902782 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:50.902790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:50.902793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:50.902942 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:50.903324 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:50.903328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:50.903330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:50.903728 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:50.904007 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:50.904326 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 02:13:50.904574 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:13:50.905638 1257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:50.993085 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 1135
I20260926 02:13:51.012434 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 1267
Sep 26 02:13:51 dist-test-slave-hllj krb5kdc[897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:13:51.030712 1409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:51.013230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.28.152.131:46439)
0926 02:13:51.013386 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:51.013390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:51.013445 (+ 55us) server_negotiation.cc:407] Connection header received
0926 02:13:51.013639 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:51.013644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:51.013736 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:51.013815 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:51.015480 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:51.016228 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:51.018056 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:51.018503 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:51.018594 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:51.024304 (+ 5710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:51.024331 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:51.024348 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:13:51.024396 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:51.027404 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:51.028390 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:51.028402 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:51.028404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:51.028570 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:13:51.029048 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:13:51.029052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:13:51.029054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:13:51.029487 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:51.029719 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:51.030155 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 02:13:51.030345 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:13:51.031479 1389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:51.149493 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 1267
I20260926 02:13:51.165210 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 910
I20260926 02:13:51.166337 971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:51.271397 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 910
2026-09-26T02:13:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4058 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T02:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:51Z Disabled control of system clock
I20260926 02:13:51.333611 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:44173
--webserver_interface=127.28.152.190
--webserver_port=0
--builtin_ntp_servers=127.28.152.148:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:44173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 02:13:51.660392 1446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:51.660717 1446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:51.660837 1446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:51.668254 1446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:51.668406 1446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:51.668473 1446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:51.668550 1446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:51.668602 1446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:13:51.678022 1446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:44173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:51.679409 1446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:51.681331 1446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:51.693519 1452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:51.693548 1454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:51.693602 1451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:51.694095 1446 server_base.cc:1034] running on GCE node
I20260926 02:13:51.695083 1446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:51.696547 1446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:51.697777 1446 hybrid_clock.cc:648] HybridClock initialized: now 1790388831697727 us; error 37 us; skew 500 ppm
I20260926 02:13:51.698153 1446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:13:51.716315 1446 webserver.cc:460] Webserver started at http://127.28.152.190:36839/ using document root <none> and password file <none>
I20260926 02:13:51.717207 1446 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:51.717336 1446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:51.717684 1446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:51.720800 1446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "7e0269b3908246478d5db0c60f1b818c"
format_stamp: "Formatted at 2026-09-26 02:13:51 on dist-test-slave-hllj"
server_key: "34aa47fd7d13929d9df2ffe9525c02b5"
server_key_iv: "184b586cddcdf20ecf52520f0f62b5aa"
server_key_version: "encryptionkey@0"
I20260926 02:13:51.721654 1446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "7e0269b3908246478d5db0c60f1b818c"
format_stamp: "Formatted at 2026-09-26 02:13:51 on dist-test-slave-hllj"
server_key: "34aa47fd7d13929d9df2ffe9525c02b5"
server_key_iv: "184b586cddcdf20ecf52520f0f62b5aa"
server_key_version: "encryptionkey@0"
I20260926 02:13:51.727861 1446 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:13:51.731442 1460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:51.732928 1446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 02:13:51.733096 1446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "7e0269b3908246478d5db0c60f1b818c"
format_stamp: "Formatted at 2026-09-26 02:13:51 on dist-test-slave-hllj"
server_key: "34aa47fd7d13929d9df2ffe9525c02b5"
server_key_iv: "184b586cddcdf20ecf52520f0f62b5aa"
server_key_version: "encryptionkey@0"
I20260926 02:13:51.733342 1446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:51.739404 1446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:51.740299 1446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:51.740597 1446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:51.758221 1446 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:44173
I20260926 02:13:51.758255 1511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:44173 every 8 connection(s)
I20260926 02:13:51.759650 1446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:51.763113 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 1446
I20260926 02:13:51.763352 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:51.763908 1512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:51.764109 29282 external_mini_cluster.cc:1428] Setting key 1e806dd75739b8b7b7d8d5c37876289f
I20260926 02:13:51.772405 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:51.766257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50092 (local address 127.28.152.190:44173)
0926 02:13:51.767318 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:51.767330 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:51.767378 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:13:51.768828 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:51.768851 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:51.769365 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:51.769719 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:51.770784 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:51.770790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:51.770803 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:51.770816 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:51.770885 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:51.770903 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:51.771117 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:51.771215 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 02:13:51.771493 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":365,"threads_started":1}
I20260926 02:13:51.776851 1512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c: Bootstrap starting.
I20260926 02:13:51.779827 1512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:51.780848 1512 log.cc:824] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:51.783607 1512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c: No bootstrap required, opened a new log
I20260926 02:13:51.790660 1512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0269b3908246478d5db0c60f1b818c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44173 } }
I20260926 02:13:51.791059 1512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:51.791162 1512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0269b3908246478d5db0c60f1b818c, State: Initialized, Role: FOLLOWER
I20260926 02:13:51.791776 1512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0269b3908246478d5db0c60f1b818c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44173 } }
I20260926 02:13:51.792024 1512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:51.792164 1512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:51.792301 1512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:51.794494 1512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0269b3908246478d5db0c60f1b818c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44173 } }
I20260926 02:13:51.794989 1512 leader_election.cc:302] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e0269b3908246478d5db0c60f1b818c; no voters:
I20260926 02:13:51.795549 1512 leader_election.cc:288] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:51.795703 1517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:51.796368 1517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [term 1 LEADER]: Becoming Leader. State: Replica: 7e0269b3908246478d5db0c60f1b818c, State: Running, Role: LEADER
I20260926 02:13:51.796947 1517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0269b3908246478d5db0c60f1b818c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44173 } }
I20260926 02:13:51.797686 1512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:51.800464 1519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0269b3908246478d5db0c60f1b818c. Latest consensus state: current_term: 1 leader_uuid: "7e0269b3908246478d5db0c60f1b818c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0269b3908246478d5db0c60f1b818c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44173 } } }
I20260926 02:13:51.800974 1519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:51.800570 1518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e0269b3908246478d5db0c60f1b818c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0269b3908246478d5db0c60f1b818c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 44173 } } }
I20260926 02:13:51.801203 1518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:51.805742 1524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:51.810956 1524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:51.820322 1524 catalog_manager.cc:1347] Generated new cluster ID: c25e5649ba22415a9156cadad58ea910
I20260926 02:13:51.820459 1524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:51.853471 1524 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:51.853905 1524 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:51.869287 1524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c: Generated new TSK 0
I20260926 02:13:51.870476 1524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:13:51.887013 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44173
--builtin_ntp_servers=127.28.152.148:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 02:13:52.214404 1536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:52.214704 1536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:52.214826 1536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:52.222226 1536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:52.222349 1536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:52.222499 1536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:13:52.232189 1536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:52.233564 1536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:52.235381 1536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:52.248672 1544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:52.248652 1542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:52.248759 1541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:52.249471 1536 server_base.cc:1034] running on GCE node
I20260926 02:13:52.250126 1536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:52.251140 1536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:52.252363 1536 hybrid_clock.cc:648] HybridClock initialized: now 1790388832252312 us; error 33 us; skew 500 ppm
I20260926 02:13:52.252705 1536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:13:52.255188 1536 webserver.cc:460] Webserver started at http://127.28.152.129:44105/ using document root <none> and password file <none>
I20260926 02:13:52.255939 1536 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:52.256037 1536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:52.256362 1536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:52.259413 1536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "ebcc643d080a4929a92f8de8e1d012b2"
format_stamp: "Formatted at 2026-09-26 02:13:52 on dist-test-slave-hllj"
server_key: "f93dafd58115874d473671ad2e691e2f"
server_key_iv: "e5459e8d167ebfe806ceae8c7251a442"
server_key_version: "encryptionkey@0"
I20260926 02:13:52.260239 1536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "ebcc643d080a4929a92f8de8e1d012b2"
format_stamp: "Formatted at 2026-09-26 02:13:52 on dist-test-slave-hllj"
server_key: "f93dafd58115874d473671ad2e691e2f"
server_key_iv: "e5459e8d167ebfe806ceae8c7251a442"
server_key_version: "encryptionkey@0"
I20260926 02:13:52.265687 1536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:13:52.269608 1550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:52.271135 1536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:13:52.271310 1536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "ebcc643d080a4929a92f8de8e1d012b2"
format_stamp: "Formatted at 2026-09-26 02:13:52 on dist-test-slave-hllj"
server_key: "f93dafd58115874d473671ad2e691e2f"
server_key_iv: "e5459e8d167ebfe806ceae8c7251a442"
server_key_version: "encryptionkey@0"
I20260926 02:13:52.271545 1536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:52.277220 1536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:52.277999 1536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:52.278214 1536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:52.279065 1536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:52.280481 1536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:52.280563 1536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:52.280644 1536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:52.280694 1536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:52.310520 1536 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:33799
I20260926 02:13:52.310570 1662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:33799 every 8 connection(s)
I20260926 02:13:52.311815 1536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:13:52.315173 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 1536
I20260926 02:13:52.315423 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:13:52.316174 29282 external_mini_cluster.cc:1428] Setting key d31785ffab3fad676d1c5b8704433405
I20260926 02:13:52.318943 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44173
--builtin_ntp_servers=127.28.152.148:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 02:13:52.320152 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:52.314322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:49325 (local address 127.28.152.190:44173)
0926 02:13:52.314737 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:52.314742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:52.316075 (+ 1333us) server_negotiation.cc:407] Connection header received
0926 02:13:52.317957 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:52.317962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:52.318037 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:52.318123 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:52.319424 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:52.319430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:52.319435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:52.319453 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:52.319492 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:52.319503 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:52.319607 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:52.319790 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 02:13:52.319855 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":209,"threads_started":1}
I20260926 02:13:52.321215 1666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:52.314921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:44173 (local address 127.28.152.129:49325)
0926 02:13:52.315913 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:52.315960 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:52.317674 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:52.318386 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:52.318395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:52.318494 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:52.319139 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:13:52.319142 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:52.319185 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:13:52.319273 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:52.319583 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:52.319595 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:52.319602 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:13:52.319865 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 02:13:52.320143 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":322,"threads_started":1}
I20260926 02:13:52.326236 1663 heartbeater.cc:347] Connected to a master server at 127.28.152.190:44173
I20260926 02:13:52.326625 1663 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:52.327382 1663 heartbeater.cc:511] Master 127.28.152.190:44173 requested a full tablet report, sending...
I20260926 02:13:52.330154 1477 ts_manager.cc:194] Registered new tserver with Master: ebcc643d080a4929a92f8de8e1d012b2 (127.28.152.129:33799)
W20260926 02:13:52.644184 1667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:52.644467 1667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:52.644559 1667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:52.652371 1667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:52.652483 1667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:52.652591 1667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:13:52.661864 1667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:52.663120 1667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:52.664942 1667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:52.678401 1672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:52.678444 1673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:52.678366 1675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:52.679351 1667 server_base.cc:1034] running on GCE node
I20260926 02:13:52.680110 1667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:52.681123 1667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:52.682382 1667 hybrid_clock.cc:648] HybridClock initialized: now 1790388832682327 us; error 51 us; skew 500 ppm
I20260926 02:13:52.682709 1667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:13:52.684962 1667 webserver.cc:460] Webserver started at http://127.28.152.130:38311/ using document root <none> and password file <none>
I20260926 02:13:52.685726 1667 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:52.685845 1667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:52.686179 1667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:52.689054 1667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "dd9867d4bb9340fe84a4f275c7eee87f"
format_stamp: "Formatted at 2026-09-26 02:13:52 on dist-test-slave-hllj"
server_key: "81b0d784a756c4284af7a39b679bd88f"
server_key_iv: "c1bad68a41653164dce3804724b1b5e1"
server_key_version: "encryptionkey@0"
I20260926 02:13:52.689963 1667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "dd9867d4bb9340fe84a4f275c7eee87f"
format_stamp: "Formatted at 2026-09-26 02:13:52 on dist-test-slave-hllj"
server_key: "81b0d784a756c4284af7a39b679bd88f"
server_key_iv: "c1bad68a41653164dce3804724b1b5e1"
server_key_version: "encryptionkey@0"
I20260926 02:13:52.695132 1667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 02:13:52.698578 1681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:52.699960 1667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:52.700127 1667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "dd9867d4bb9340fe84a4f275c7eee87f"
format_stamp: "Formatted at 2026-09-26 02:13:52 on dist-test-slave-hllj"
server_key: "81b0d784a756c4284af7a39b679bd88f"
server_key_iv: "c1bad68a41653164dce3804724b1b5e1"
server_key_version: "encryptionkey@0"
I20260926 02:13:52.700323 1667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:52.706012 1667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:52.706826 1667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:52.707072 1667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:52.707974 1667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:52.709549 1667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:52.709636 1667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:52.709717 1667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:52.709770 1667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:52.745044 1667 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:46149
I20260926 02:13:52.745105 1793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:46149 every 8 connection(s)
I20260926 02:13:52.746371 1667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:13:52.751240 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 1667
I20260926 02:13:52.751395 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:13:52.751807 29282 external_mini_cluster.cc:1428] Setting key ab9afdae8d7cee0260dd89b14db1f2a5
I20260926 02:13:52.754647 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44173
--builtin_ntp_servers=127.28.152.148:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 02:13:52.754765 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:52.748863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:34719 (local address 127.28.152.190:44173)
0926 02:13:52.749041 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:52.749046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:52.750597 (+ 1551us) server_negotiation.cc:407] Connection header received
0926 02:13:52.752379 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:52.752383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:52.752450 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:52.752556 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:52.754076 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:52.754081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:52.754086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:52.754105 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:52.754136 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:52.754147 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:52.754243 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:52.754438 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 02:13:52.754504 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:13:52.755756 1796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:52.749474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:44173 (local address 127.28.152.130:34719)
0926 02:13:52.750430 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:52.750466 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:52.752110 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:52.752768 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:52.752779 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:52.752925 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:52.753680 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:13:52.753683 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:52.753747 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:13:52.753869 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:52.754236 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:52.754246 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:52.754254 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:13:52.754509 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 02:13:52.754788 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":258,"threads_started":1}
I20260926 02:13:52.757118 1794 heartbeater.cc:347] Connected to a master server at 127.28.152.190:44173
I20260926 02:13:52.757486 1794 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:52.758224 1794 heartbeater.cc:511] Master 127.28.152.190:44173 requested a full tablet report, sending...
I20260926 02:13:52.759796 1477 ts_manager.cc:194] Registered new tserver with Master: dd9867d4bb9340fe84a4f275c7eee87f (127.28.152.130:46149)
W20260926 02:13:53.085933 1797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:53.086216 1797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:53.086325 1797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:53.093963 1797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:53.094062 1797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:53.094194 1797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:13:53.103626 1797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:44173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:53.104990 1797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:53.106781 1797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:53.119621 1802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:53.119587 1803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:53.119642 1805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:53.122592 1797 server_base.cc:1034] running on GCE node
I20260926 02:13:53.123301 1797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:53.124296 1797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:53.125502 1797 hybrid_clock.cc:648] HybridClock initialized: now 1790388833125434 us; error 55 us; skew 500 ppm
I20260926 02:13:53.125859 1797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:13:53.128197 1797 webserver.cc:460] Webserver started at http://127.28.152.131:41905/ using document root <none> and password file <none>
I20260926 02:13:53.128903 1797 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:53.129009 1797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:53.129383 1797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:53.132284 1797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "b920b0cc9ab54e1b807d247be488e5b4"
format_stamp: "Formatted at 2026-09-26 02:13:53 on dist-test-slave-hllj"
server_key: "be310b6fe3dc7a01b7a3c72476ec0f70"
server_key_iv: "3e7845685330b951a3ae7791d24cea8b"
server_key_version: "encryptionkey@0"
I20260926 02:13:53.133056 1797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "b920b0cc9ab54e1b807d247be488e5b4"
format_stamp: "Formatted at 2026-09-26 02:13:53 on dist-test-slave-hllj"
server_key: "be310b6fe3dc7a01b7a3c72476ec0f70"
server_key_iv: "3e7845685330b951a3ae7791d24cea8b"
server_key_version: "encryptionkey@0"
I20260926 02:13:53.138375 1797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:13:53.141798 1811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:53.143110 1797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:53.143254 1797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "b920b0cc9ab54e1b807d247be488e5b4"
format_stamp: "Formatted at 2026-09-26 02:13:53 on dist-test-slave-hllj"
server_key: "be310b6fe3dc7a01b7a3c72476ec0f70"
server_key_iv: "3e7845685330b951a3ae7791d24cea8b"
server_key_version: "encryptionkey@0"
I20260926 02:13:53.143452 1797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:53.149122 1797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:53.150018 1797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:53.150265 1797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:53.151151 1797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:13:53.152521 1797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:13:53.152613 1797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:53.152714 1797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:13:53.152781 1797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:53.181782 1797 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:36897
I20260926 02:13:53.181833 1923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:36897 every 8 connection(s)
I20260926 02:13:53.183091 1797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:13:53.186450 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 1797
I20260926 02:13:53.186610 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:13:53.187088 29282 external_mini_cluster.cc:1428] Setting key 941b2145c9f6502b9d89ed0e5cc6255a
I20260926 02:13:53.191301 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.185552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:33179 (local address 127.28.152.190:44173)
0926 02:13:53.185721 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:53.185725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:53.187315 (+ 1590us) server_negotiation.cc:407] Connection header received
0926 02:13:53.189174 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:53.189179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:53.189247 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:53.189342 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:53.190758 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:53.190762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:53.190765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:53.190781 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:53.190814 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:53.190825 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:53.190908 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:53.191033 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 02:13:53.191080 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:13:53.192245 1926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.186131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:44173 (local address 127.28.152.131:33179)
0926 02:13:53.187157 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.187202 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:53.188909 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:53.189717 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:53.189725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:53.189823 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:53.190486 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:13:53.190489 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:53.190528 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:13:53.190612 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:53.190889 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:53.190894 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:53.190899 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:13:53.191095 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 02:13:53.191297 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":299,"threads_started":1}
I20260926 02:13:53.193514 1924 heartbeater.cc:347] Connected to a master server at 127.28.152.190:44173
I20260926 02:13:53.193796 1924 heartbeater.cc:464] Registering TS with master...
I20260926 02:13:53.194391 1924 heartbeater.cc:511] Master 127.28.152.190:44173 requested a full tablet report, sending...
I20260926 02:13:53.195719 1477 ts_manager.cc:194] Registered new tserver with Master: b920b0cc9ab54e1b807d247be488e5b4 (127.28.152.131:36897)
I20260926 02:13:53.202250 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:13:53.208940 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.206669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.28.152.190:44173)
0926 02:13:53.206826 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:53.206831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:53.207365 (+ 534us) server_negotiation.cc:407] Connection header received
0926 02:13:53.207500 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:53.207520 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:53.207574 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:53.207653 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:53.208471 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:53.208476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:53.208479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:53.208494 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:53.208541 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:53.208552 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:53.208645 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:53.208696 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 02:13:53.208739 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:13:53.212693 1477 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:13:53.219296 1477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:13:53.236773 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.230921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:33799 (local address 127.0.0.1:50916)
0926 02:13:53.232357 (+ 1436us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.232403 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:53.232549 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:53.234415 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:53.234424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:53.234536 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:53.235394 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:13:53.235398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:53.235431 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:13:53.235526 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:53.236042 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:53.236053 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:53.236061 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:13:53.236231 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 02:13:53.236344 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"threads_started":1}
I20260926 02:13:53.237758 1937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.231058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50916 (local address 127.28.152.129:33799)
0926 02:13:53.231633 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:53.231643 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:53.232704 (+ 1061us) server_negotiation.cc:407] Connection header received
0926 02:13:53.232783 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:53.232793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:53.233119 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:53.233307 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:53.235707 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:53.235717 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:53.235734 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:53.235764 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:53.235828 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:53.235851 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:53.235978 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:53.237303 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0926 02:13:53.237418 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":277,"threads_started":1}
I20260926 02:13:53.240566 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.231536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:46149 (local address 127.0.0.1:44718)
0926 02:13:53.236966 (+ 5430us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.236982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:53.237062 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:53.238108 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:53.238112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:53.238134 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:53.238678 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:13:53.238682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:53.238698 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:13:53.238743 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:53.240057 (+ 1314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:53.240063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:53.240070 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:13:53.240192 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 02:13:53.240259 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5346,"thread_start_us":3740,"threads_started":1}
I20260926 02:13:53.240751 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.231389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:36897 (local address 127.0.0.1:42684)
0926 02:13:53.236004 (+ 4615us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.236023 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:53.236108 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:53.237297 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:53.237302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:53.237324 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:53.238682 (+ 1358us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:13:53.238685 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:53.238698 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:13:53.238743 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:53.240125 (+ 1382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:53.240130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:53.240136 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:13:53.240264 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 02:13:53.240322 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4467,"thread_start_us":885,"threads_started":1}
I20260926 02:13:53.240962 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.232123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.28.152.130:46149)
0926 02:13:53.237568 (+ 5445us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:53.237576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:53.237606 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:13:53.237673 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:53.237681 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:53.237914 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:53.238395 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:53.239811 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:53.239819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:53.239833 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:53.239857 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:53.239910 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:53.239929 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:53.240114 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:53.240516 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 02:13:53.240621 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5277,"thread_start_us":5147,"threads_started":1}
I20260926 02:13:53.241776 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.231915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.28.152.131:36897)
0926 02:13:53.236668 (+ 4753us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:53.236677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:53.236713 (+ 36us) server_negotiation.cc:407] Connection header received
0926 02:13:53.236783 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:53.236792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:53.237102 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:53.238904 (+ 1802us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:53.239913 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:53.239920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:53.239929 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:53.239945 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:53.239988 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:53.240007 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:53.241233 (+ 1226us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:53.241296 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 02:13:53.241392 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4574,"thread_start_us":4411,"threads_started":1}
I20260926 02:13:53.244267 1598 tablet_service.cc:1467] Processing CreateTablet for tablet 709fb3ba7f26440fb3af6c7a2a459c3f (DEFAULT_TABLE table=test-table [id=b0ed75d9d65a48e492b2bbc850ba833a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:53.244706 1729 tablet_service.cc:1467] Processing CreateTablet for tablet 709fb3ba7f26440fb3af6c7a2a459c3f (DEFAULT_TABLE table=test-table [id=b0ed75d9d65a48e492b2bbc850ba833a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:53.245188 1859 tablet_service.cc:1467] Processing CreateTablet for tablet 709fb3ba7f26440fb3af6c7a2a459c3f (DEFAULT_TABLE table=test-table [id=b0ed75d9d65a48e492b2bbc850ba833a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:13:53.245806 1598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709fb3ba7f26440fb3af6c7a2a459c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:53.246235 1729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709fb3ba7f26440fb3af6c7a2a459c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:53.246608 1859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709fb3ba7f26440fb3af6c7a2a459c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:53.259671 1942 tablet_bootstrap.cc:492] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2: Bootstrap starting.
I20260926 02:13:53.261823 1943 tablet_bootstrap.cc:492] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f: Bootstrap starting.
I20260926 02:13:53.263053 1942 tablet_bootstrap.cc:654] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:53.263616 1944 tablet_bootstrap.cc:492] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4: Bootstrap starting.
I20260926 02:13:53.264349 1942 log.cc:824] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:53.265602 1943 tablet_bootstrap.cc:654] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:53.266814 1943 log.cc:824] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:53.266979 1944 tablet_bootstrap.cc:654] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:53.268144 1944 log.cc:824] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:53.268714 1942 tablet_bootstrap.cc:492] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2: No bootstrap required, opened a new log
I20260926 02:13:53.269003 1942 ts_tablet_manager.cc:1397] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260926 02:13:53.269800 1943 tablet_bootstrap.cc:492] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f: No bootstrap required, opened a new log
I20260926 02:13:53.270068 1943 ts_tablet_manager.cc:1397] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 02:13:53.270696 1944 tablet_bootstrap.cc:492] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4: No bootstrap required, opened a new log
I20260926 02:13:53.270923 1944 ts_tablet_manager.cc:1397] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 02:13:53.278532 1944 raft_consensus.cc:348] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcc643d080a4929a92f8de8e1d012b2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 33799 } } peers { permanent_uuid: "b920b0cc9ab54e1b807d247be488e5b4" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36897 } } peers { permanent_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46149 } }
I20260926 02:13:53.278946 1944 raft_consensus.cc:374] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:53.279053 1944 raft_consensus.cc:729] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b920b0cc9ab54e1b807d247be488e5b4, State: Initialized, Role: FOLLOWER
I20260926 02:13:53.279670 1944 consensus_queue.cc:260] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcc643d080a4929a92f8de8e1d012b2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 33799 } } peers { permanent_uuid: "b920b0cc9ab54e1b807d247be488e5b4" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36897 } } peers { permanent_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46149 } }
I20260926 02:13:53.279655 1942 raft_consensus.cc:348] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcc643d080a4929a92f8de8e1d012b2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 33799 } } peers { permanent_uuid: "b920b0cc9ab54e1b807d247be488e5b4" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36897 } } peers { permanent_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46149 } }
I20260926 02:13:53.280097 1942 raft_consensus.cc:374] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:53.280205 1942 raft_consensus.cc:729] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcc643d080a4929a92f8de8e1d012b2, State: Initialized, Role: FOLLOWER
I20260926 02:13:53.280894 1924 heartbeater.cc:503] Master 127.28.152.190:44173 was elected leader, sending a full tablet report...
I20260926 02:13:53.280925 1942 consensus_queue.cc:260] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcc643d080a4929a92f8de8e1d012b2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 33799 } } peers { permanent_uuid: "b920b0cc9ab54e1b807d247be488e5b4" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36897 } } peers { permanent_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46149 } }
I20260926 02:13:53.281596 1944 ts_tablet_manager.cc:1428] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260926 02:13:53.280506 1943 raft_consensus.cc:348] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcc643d080a4929a92f8de8e1d012b2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 33799 } } peers { permanent_uuid: "b920b0cc9ab54e1b807d247be488e5b4" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36897 } } peers { permanent_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46149 } }
I20260926 02:13:53.282135 1943 raft_consensus.cc:374] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:53.282235 1943 raft_consensus.cc:729] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd9867d4bb9340fe84a4f275c7eee87f, State: Initialized, Role: FOLLOWER
I20260926 02:13:53.282841 1943 consensus_queue.cc:260] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcc643d080a4929a92f8de8e1d012b2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 33799 } } peers { permanent_uuid: "b920b0cc9ab54e1b807d247be488e5b4" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36897 } } peers { permanent_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46149 } }
I20260926 02:13:53.283232 1663 heartbeater.cc:503] Master 127.28.152.190:44173 was elected leader, sending a full tablet report...
I20260926 02:13:53.283967 1794 heartbeater.cc:503] Master 127.28.152.190:44173 was elected leader, sending a full tablet report...
I20260926 02:13:53.284624 1943 ts_tablet_manager.cc:1428] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260926 02:13:53.286643 1942 ts_tablet_manager.cc:1428] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
W20260926 02:13:53.313975 1664 tablet.cc:2369] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:13:53.359269 1948 raft_consensus.cc:482] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:13:53.359576 1948 raft_consensus.cc:504] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcc643d080a4929a92f8de8e1d012b2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 33799 } } peers { permanent_uuid: "b920b0cc9ab54e1b807d247be488e5b4" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36897 } } peers { permanent_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46149 } }
I20260926 02:13:53.361295 1948 leader_election.cc:288] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebcc643d080a4929a92f8de8e1d012b2 (127.28.152.129:33799), dd9867d4bb9340fe84a4f275c7eee87f (127.28.152.130:46149)
I20260926 02:13:53.363862 1926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.361592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:46149 (local address 127.28.152.131:57001)
0926 02:13:53.361779 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.361799 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:53.361903 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:53.362380 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:53.362385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:53.362410 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:53.363085 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:13:53.363090 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:53.363112 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:13:53.363152 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:53.363453 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:53.363458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:53.363463 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:13:53.363561 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 02:13:53.363615 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 02:13:53.363855 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.361901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:57001 (local address 127.28.152.130:46149)
0926 02:13:53.362122 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:53.362126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:53.362141 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:13:53.362189 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:53.362193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:53.362236 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:53.362317 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:53.363307 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:53.363311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:53.363314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:53.363328 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:53.363357 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:53.363368 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:53.363473 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:53.363580 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 02:13:53.363631 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 02:13:53.364259 1937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.361844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:48297 (local address 127.28.152.129:33799)
0926 02:13:53.362017 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:53.362036 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:53.362559 (+ 523us) server_negotiation.cc:407] Connection header received
0926 02:13:53.362627 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:53.362632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:53.362690 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:53.362764 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:53.363679 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:53.363684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:53.363687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:53.363703 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:53.363731 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:53.363742 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:53.363838 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:53.363943 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 02:13:53.364000 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:13:53.364423 1951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.361675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:33799 (local address 127.28.152.131:48297)
0926 02:13:53.362430 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.362445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:53.362540 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:53.362837 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:53.362842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:53.362864 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:13:53.363477 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:13:53.363479 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:13:53.363492 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:13:53.363528 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:13:53.363822 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:13:53.363827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:13:53.363831 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 02:13:53.363946 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 02:13:53.363993 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":455,"threads_started":1}
I20260926 02:13:53.364822 1749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709fb3ba7f26440fb3af6c7a2a459c3f" candidate_uuid: "b920b0cc9ab54e1b807d247be488e5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" is_pre_election: true
I20260926 02:13:53.365413 1749 raft_consensus.cc:2454] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b920b0cc9ab54e1b807d247be488e5b4 in term 0.
I20260926 02:13:53.365381 1618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709fb3ba7f26440fb3af6c7a2a459c3f" candidate_uuid: "b920b0cc9ab54e1b807d247be488e5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebcc643d080a4929a92f8de8e1d012b2" is_pre_election: true
I20260926 02:13:53.365959 1618 raft_consensus.cc:2454] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b920b0cc9ab54e1b807d247be488e5b4 in term 0.
I20260926 02:13:53.366133 1813 leader_election.cc:302] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b920b0cc9ab54e1b807d247be488e5b4, dd9867d4bb9340fe84a4f275c7eee87f; no voters:
I20260926 02:13:53.366549 1948 raft_consensus.cc:2781] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:13:53.366690 1948 raft_consensus.cc:482] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:13:53.366778 1948 raft_consensus.cc:3037] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:53.368899 1948 raft_consensus.cc:504] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcc643d080a4929a92f8de8e1d012b2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 33799 } } peers { permanent_uuid: "b920b0cc9ab54e1b807d247be488e5b4" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36897 } } peers { permanent_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46149 } }
I20260926 02:13:53.369562 1948 leader_election.cc:288] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [CANDIDATE]: Term 1 election: Requested vote from peers ebcc643d080a4929a92f8de8e1d012b2 (127.28.152.129:33799), dd9867d4bb9340fe84a4f275c7eee87f (127.28.152.130:46149)
I20260926 02:13:53.370014 1618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709fb3ba7f26440fb3af6c7a2a459c3f" candidate_uuid: "b920b0cc9ab54e1b807d247be488e5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebcc643d080a4929a92f8de8e1d012b2"
I20260926 02:13:53.370203 1618 raft_consensus.cc:3037] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:53.373222 1618 raft_consensus.cc:2454] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b920b0cc9ab54e1b807d247be488e5b4 in term 1.
I20260926 02:13:53.373387 1749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709fb3ba7f26440fb3af6c7a2a459c3f" candidate_uuid: "b920b0cc9ab54e1b807d247be488e5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd9867d4bb9340fe84a4f275c7eee87f"
I20260926 02:13:53.373637 1749 raft_consensus.cc:3037] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:53.373658 1815 leader_election.cc:302] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b920b0cc9ab54e1b807d247be488e5b4, ebcc643d080a4929a92f8de8e1d012b2; no voters:
I20260926 02:13:53.373986 1948 raft_consensus.cc:2781] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:53.374876 1948 raft_consensus.cc:686] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 1 LEADER]: Becoming Leader. State: Replica: b920b0cc9ab54e1b807d247be488e5b4, State: Running, Role: LEADER
I20260926 02:13:53.375456 1948 consensus_queue.cc:237] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcc643d080a4929a92f8de8e1d012b2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 33799 } } peers { permanent_uuid: "b920b0cc9ab54e1b807d247be488e5b4" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36897 } } peers { permanent_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46149 } }
I20260926 02:13:53.376398 1749 raft_consensus.cc:2454] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b920b0cc9ab54e1b807d247be488e5b4 in term 1.
I20260926 02:13:53.380625 1476 catalog_manager.cc:5495] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b920b0cc9ab54e1b807d247be488e5b4 (127.28.152.131). New cstate: current_term: 1 leader_uuid: "b920b0cc9ab54e1b807d247be488e5b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcc643d080a4929a92f8de8e1d012b2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 33799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b920b0cc9ab54e1b807d247be488e5b4" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20260926 02:13:53.398901 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.396929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.28.152.131:36897)
0926 02:13:53.397098 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:53.397102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:53.397291 (+ 189us) server_negotiation.cc:407] Connection header received
0926 02:13:53.397489 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:53.397493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:53.397576 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:53.397676 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:53.398449 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:53.398455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:53.398458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:53.398471 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:53.398497 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:53.398508 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:53.398601 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:53.398660 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 02:13:53.398708 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 02:13:53.407388 1749 raft_consensus.cc:1261] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f [term 1 FOLLOWER]: Refusing update from remote peer b920b0cc9ab54e1b807d247be488e5b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:53.407388 1618 raft_consensus.cc:1261] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2 [term 1 FOLLOWER]: Refusing update from remote peer b920b0cc9ab54e1b807d247be488e5b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:13:53.408373 1952 consensus_queue.cc:1035] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd9867d4bb9340fe84a4f275c7eee87f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:53.408733 1948 consensus_queue.cc:1035] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebcc643d080a4929a92f8de8e1d012b2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:13:53.421825 1957 mvcc.cc:205] Tried to move back new op lower bound from 7333432661626302464 to 7333432661510705152. Current Snapshot: MvccSnapshot[applied={T|T < 7333432661626302464}]
I20260926 02:13:53.432544 1476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:53.432821 1476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:13:53.436474 1476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e0269b3908246478d5db0c60f1b818c: Sending DeleteTablet for 3 replicas of tablet 709fb3ba7f26440fb3af6c7a2a459c3f
I20260926 02:13:53.437793 1859 tablet_service.cc:1514] Processing DeleteTablet for tablet 709fb3ba7f26440fb3af6c7a2a459c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:53 UTC) from {username='slave'} at 127.0.0.1:42684
I20260926 02:13:53.437929 1729 tablet_service.cc:1514] Processing DeleteTablet for tablet 709fb3ba7f26440fb3af6c7a2a459c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:53 UTC) from {username='slave'} at 127.0.0.1:44718
I20260926 02:13:53.437937 1598 tablet_service.cc:1514] Processing DeleteTablet for tablet 709fb3ba7f26440fb3af6c7a2a459c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:13:53 UTC) from {username='slave'} at 127.0.0.1:50916
I20260926 02:13:53.440497 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 1536
I20260926 02:13:53.445665 1965 tablet_replica.cc:330] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f: stopping tablet replica
I20260926 02:13:53.446188 1965 raft_consensus.cc:2229] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:53.446743 1965 raft_consensus.cc:2258] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:53.447539 1966 tablet_replica.cc:330] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2: stopping tablet replica
I20260926 02:13:53.447994 1966 raft_consensus.cc:2229] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:13:53.448503 1966 raft_consensus.cc:2258] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:53.451352 1965 ts_tablet_manager.cc:1905] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:53.456133 1964 tablet_replica.cc:330] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4: stopping tablet replica
I20260926 02:13:53.456658 1964 raft_consensus.cc:2229] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:13:53.456745 1937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.441940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.28.152.129:33799)
0926 02:13:53.442114 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:53.442121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:53.454029 (+ 11908us) server_negotiation.cc:407] Connection header received
0926 02:13:53.454546 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:53.454554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:53.454654 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:53.454733 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:53.455926 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:53.455933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:53.455937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:53.455959 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:53.456001 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:53.456018 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:53.456156 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:53.456368 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 02:13:53.456437 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:13:53.457773 1658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:53.459297 1964 raft_consensus.cc:2258] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:13:53.459396 1965 ts_tablet_manager.cc:1918] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:13:53.459720 1965 log.cc:1196] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/709fb3ba7f26440fb3af6c7a2a459c3f
I20260926 02:13:53.460327 1965 ts_tablet_manager.cc:1939] T 709fb3ba7f26440fb3af6c7a2a459c3f P dd9867d4bb9340fe84a4f275c7eee87f: Deleting consensus metadata
I20260926 02:13:53.461685 1964 ts_tablet_manager.cc:1905] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:53.462751 1462 catalog_manager.cc:4841] TS dd9867d4bb9340fe84a4f275c7eee87f (127.28.152.130:46149): tablet 709fb3ba7f26440fb3af6c7a2a459c3f (table test-table [id=b0ed75d9d65a48e492b2bbc850ba833a]) successfully deleted
I20260926 02:13:53.469789 1964 ts_tablet_manager.cc:1918] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:13:53.470000 1964 log.cc:1196] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/709fb3ba7f26440fb3af6c7a2a459c3f
I20260926 02:13:53.470919 1964 ts_tablet_manager.cc:1939] T 709fb3ba7f26440fb3af6c7a2a459c3f P b920b0cc9ab54e1b807d247be488e5b4: Deleting consensus metadata
I20260926 02:13:53.473385 1463 catalog_manager.cc:4841] TS b920b0cc9ab54e1b807d247be488e5b4 (127.28.152.131:36897): tablet 709fb3ba7f26440fb3af6c7a2a459c3f (table test-table [id=b0ed75d9d65a48e492b2bbc850ba833a]) successfully deleted
I20260926 02:13:53.458695 1966 ts_tablet_manager.cc:1905] T 709fb3ba7f26440fb3af6c7a2a459c3f P ebcc643d080a4929a92f8de8e1d012b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:13:53.584232 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 1536
I20260926 02:13:53.605309 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 1667
W20260926 02:13:53.605405 1464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.152.129:33799 (error 108)
I20260926 02:13:53.608069 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.607556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:33799 (local address 127.0.0.1:50924)
0926 02:13:53.607745 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.607884 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:33799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 02:13:53.608311 1464 catalog_manager.cc:4584] TS ebcc643d080a4929a92f8de8e1d012b2 (127.28.152.129:33799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 709fb3ba7f26440fb3af6c7a2a459c3f: Network error: Client connection negotiation failed: client connection to 127.28.152.129:33799: connect: Connection refused (error 111)
I20260926 02:13:53.609331 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.606166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.28.152.130:46149)
0926 02:13:53.606346 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:53.606353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:53.606715 (+ 362us) server_negotiation.cc:407] Connection header received
0926 02:13:53.606793 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:53.606798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:53.606900 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:53.606989 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:53.608611 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:53.608618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:53.608623 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:53.608646 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:53.608688 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:53.608703 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:53.608826 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:53.608949 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 02:13:53.609011 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:13:53.610010 1789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:53.662357 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.661779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:33799 (local address 127.0.0.1:50938)
0926 02:13:53.662037 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.662163 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:33799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 02:13:53.663594 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.663199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:33799 (local address 127.0.0.1:50944)
0926 02:13:53.663356 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.663455 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:33799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 02:13:53.695708 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 1667
I20260926 02:13:53.709240 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 1797
I20260926 02:13:53.712225 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.710158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.28.152.131:36897)
0926 02:13:53.710312 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:53.710317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:53.710337 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:13:53.710449 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:53.710453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:53.710505 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:53.710588 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:53.711540 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:53.711545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:53.711548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:53.711563 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:53.711596 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:53.711611 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:53.711718 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:53.711861 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 02:13:53.711914 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:13:53.712886 1919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:53.724116 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.723613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:33799 (local address 127.0.0.1:50958)
0926 02:13:53.723820 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.723955 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:33799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:13:53.725263 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.724871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:33799 (local address 127.0.0.1:50974)
0926 02:13:53.725006 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.725089 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:33799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 02:13:53.805968 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.805413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:33799 (local address 127.0.0.1:50982)
0926 02:13:53.805681 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.805787 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:33799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 02:13:53.807060 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:53.806664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:33799 (local address 127.0.0.1:50994)
0926 02:13:53.806860 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:53.806937 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:33799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 02:13:53.833989 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 1797
I20260926 02:13:53.850140 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 1446
I20260926 02:13:53.851333 1507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:53.954339 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 1446
2026-09-26T02:13:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2682 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 02:13:53.991631 29282 mini_oidc.cc:152] Starting JWKS server
I20260926 02:13:53.994364 29282 webserver.cc:460] Webserver started at https://127.0.0.1:46821/ using document root <none> and password file <none>
I20260926 02:13:53.994681 29282 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:13:53.995350 29282 webserver.cc:460] Webserver started at http://127.0.0.1:34727/ using document root <none> and password file <none>
2026-09-26T02:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:54Z Disabled control of system clock
I20260926 02:13:54.021245 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:40873
--webserver_interface=127.28.152.190
--webserver_port=0
--builtin_ntp_servers=127.28.152.148:38825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:40873
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46821/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/testchainca.cert with env {}
W20260926 02:13:54.347196 1978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:54.347508 1978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:54.347599 1978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:54.354969 1978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:54.355093 1978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:54.355131 1978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:54.355185 1978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:54.355228 1978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:13:54.355288 1978 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:13:54.355325 1978 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46821/jwks.json
I20260926 02:13:54.365538 1978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:38825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:40873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:40873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46821/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:54.366925 1978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:54.368193 1978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:54.379968 1983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:54.379940 1986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:54.379904 1984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:54.380961 1978 server_base.cc:1034] running on GCE node
I20260926 02:13:54.381839 1978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:54.383365 1978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:54.384598 1978 hybrid_clock.cc:648] HybridClock initialized: now 1790388834384553 us; error 50 us; skew 500 ppm
I20260926 02:13:54.384955 1978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:13:54.387363 1978 webserver.cc:460] Webserver started at http://127.28.152.190:44675/ using document root <none> and password file <none>
I20260926 02:13:54.388049 1978 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:54.388159 1978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:54.388456 1978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:54.391499 1978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "5ce9adc6106f40e3aee3830e9cb9a3f1"
format_stamp: "Formatted at 2026-09-26 02:13:54 on dist-test-slave-hllj"
server_key: "ebab6acb34d0a2c0e7dab0cc37ff6536"
server_key_iv: "8d6befe8696dccaf037b2be4ee0b537d"
server_key_version: "encryptionkey@0"
I20260926 02:13:54.392309 1978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "5ce9adc6106f40e3aee3830e9cb9a3f1"
format_stamp: "Formatted at 2026-09-26 02:13:54 on dist-test-slave-hllj"
server_key: "ebab6acb34d0a2c0e7dab0cc37ff6536"
server_key_iv: "8d6befe8696dccaf037b2be4ee0b537d"
server_key_version: "encryptionkey@0"
I20260926 02:13:54.397713 1978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:13:54.401404 1992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:54.402897 1978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:54.403105 1978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "5ce9adc6106f40e3aee3830e9cb9a3f1"
format_stamp: "Formatted at 2026-09-26 02:13:54 on dist-test-slave-hllj"
server_key: "ebab6acb34d0a2c0e7dab0cc37ff6536"
server_key_iv: "8d6befe8696dccaf037b2be4ee0b537d"
server_key_version: "encryptionkey@0"
I20260926 02:13:54.403326 1978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:54.416553 1978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:54.417578 1978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:54.417901 1978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:54.436224 1978 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:40873
I20260926 02:13:54.436249 2043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:40873 every 8 connection(s)
I20260926 02:13:54.437655 1978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:54.440372 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 1978
I20260926 02:13:54.440558 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:54.441211 29282 external_mini_cluster.cc:1428] Setting key c18140e11efa88eacdf09ae61dd54f1c
I20260926 02:13:54.441670 2044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:54.455456 2044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1: Bootstrap starting.
I20260926 02:13:54.455693 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:54.443133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.28.152.190:40873)
0926 02:13:54.444106 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:54.444117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:54.444161 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:13:54.446151 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:54.446177 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:54.446738 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:54.447114 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:54.447825 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:54.449207 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:54.451047 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:54.451783 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:54.451928 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:54.453256 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:54.453266 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:54.453283 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:54.453866 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:54.453947 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:54.453968 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:54.454057 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:54.454286 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 02:13:54.454630 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":343,"threads_started":1}
I20260926 02:13:54.458824 2044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:54.459522 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:13:54.459992 2044 log.cc:824] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:54.463135 2044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1: No bootstrap required, opened a new log
I20260926 02:13:54.470635 2044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce9adc6106f40e3aee3830e9cb9a3f1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40873 } }
I20260926 02:13:54.471026 2044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:54.471127 2044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce9adc6106f40e3aee3830e9cb9a3f1, State: Initialized, Role: FOLLOWER
I20260926 02:13:54.471701 2044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce9adc6106f40e3aee3830e9cb9a3f1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40873 } }
I20260926 02:13:54.471923 2044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:54.472050 2044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:54.472203 2044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:54.474380 2044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce9adc6106f40e3aee3830e9cb9a3f1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40873 } }
I20260926 02:13:54.474929 2044 leader_election.cc:302] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ce9adc6106f40e3aee3830e9cb9a3f1; no voters:
I20260926 02:13:54.475507 2044 leader_election.cc:288] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:54.475657 2049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:54.476931 2049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce9adc6106f40e3aee3830e9cb9a3f1, State: Running, Role: LEADER
I20260926 02:13:54.477480 2049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce9adc6106f40e3aee3830e9cb9a3f1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40873 } }
I20260926 02:13:54.477898 2044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:54.480322 2051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ce9adc6106f40e3aee3830e9cb9a3f1. Latest consensus state: current_term: 1 leader_uuid: "5ce9adc6106f40e3aee3830e9cb9a3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce9adc6106f40e3aee3830e9cb9a3f1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40873 } } }
I20260926 02:13:54.480401 2050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ce9adc6106f40e3aee3830e9cb9a3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ce9adc6106f40e3aee3830e9cb9a3f1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40873 } } }
I20260926 02:13:54.480732 2051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:54.480753 2050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:54.481747 2057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:54.488052 2057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:54.498123 2057 catalog_manager.cc:1347] Generated new cluster ID: 721de5a222894e1a9536f53ff721695a
I20260926 02:13:54.498250 2057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:54.509624 2057 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:54.511030 2057 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:54.521209 2057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ce9adc6106f40e3aee3830e9cb9a3f1: Generated new TSK 0
I20260926 02:13:54.522188 2057 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 02:13:54.566264 2074 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 02:13:54.566419 2074 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:13:54.567651 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:54.545711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.28.152.190:40873)
0926 02:13:54.545918 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:54.545925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:54.546369 (+ 444us) server_negotiation.cc:407] Connection header received
0926 02:13:54.546660 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:54.546667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:54.546764 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:54.546938 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:13:54.547410 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:54.548555 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:54.550090 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:54.550775 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:54.550901 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:54.551502 (+ 601us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:13:54.566919 (+ 15417us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 02:13:54.567332 (+ 413us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43550: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 02:13:54.567773 2047 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43550: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 02:13:54.569101 2073 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:13:54.545563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:40873 (local address 127.0.0.1:43550)
0926 02:13:54.546267 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:54.546283 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:54.546439 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:54.546945 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:54.546949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:54.546958 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:13:54.547267 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:54.547275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:54.548686 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:54.548690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:54.549934 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:54.549948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:54.550958 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:54.550963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:54.551129 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:54.551202 (+ 73us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:13:54.567238 (+ 16036us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 02:13:54.567551 (+ 313us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.152.190:40873: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":373,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.152.190:40873: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46821/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 02:13:54.570794 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 1978
I20260926 02:13:54.572254 2039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:54.663820 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 1978
2026-09-26T02:13:54Z chronyd exiting
I20260926 02:13:54.710435 29282 test_util.cc:175] -----------------------------------------------
I20260926 02:13:54.710570 29282 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790388791636863-29282-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 02:13:54.714465 29282 mini_oidc.cc:152] Starting JWKS server
I20260926 02:13:54.716265 29282 webserver.cc:460] Webserver started at https://127.0.0.1:38213/ using document root <none> and password file <none>
I20260926 02:13:54.716499 29282 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:13:54.717146 29282 webserver.cc:460] Webserver started at http://127.0.0.1:35319/ using document root <none> and password file <none>
2026-09-26T02:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:54Z Disabled control of system clock
I20260926 02:13:54.743204 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:38869
--webserver_interface=127.28.152.190
--webserver_port=0
--builtin_ntp_servers=127.28.152.148:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:38869
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38213/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/testchainca.cert with env {}
W20260926 02:13:55.067021 2081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:55.067360 2081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:55.067492 2081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:55.074900 2081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:55.075037 2081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:55.075110 2081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:55.075191 2081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:55.075250 2081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:13:55.075335 2081 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:13:55.075397 2081 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38213/jwks.json
I20260926 02:13:55.085743 2081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:38869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:38869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38213/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:55.087102 2081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:55.088415 2081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:55.100116 2087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:55.100229 2086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:55.100076 2089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:55.100775 2081 server_base.cc:1034] running on GCE node
I20260926 02:13:55.101629 2081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:55.103159 2081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:55.104387 2081 hybrid_clock.cc:648] HybridClock initialized: now 1790388835104337 us; error 44 us; skew 500 ppm
I20260926 02:13:55.104759 2081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:13:55.107146 2081 webserver.cc:460] Webserver started at http://127.28.152.190:45645/ using document root <none> and password file <none>
I20260926 02:13:55.107841 2081 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:55.107934 2081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:55.108208 2081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:55.111208 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "025229d27ba44b958bac077d8833c568"
format_stamp: "Formatted at 2026-09-26 02:13:55 on dist-test-slave-hllj"
server_key: "23f4ba4fa568795ff365dafb6368365e"
server_key_iv: "af08085c7f69e65ea57ddbda748fb751"
server_key_version: "encryptionkey@0"
I20260926 02:13:55.112022 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "025229d27ba44b958bac077d8833c568"
format_stamp: "Formatted at 2026-09-26 02:13:55 on dist-test-slave-hllj"
server_key: "23f4ba4fa568795ff365dafb6368365e"
server_key_iv: "af08085c7f69e65ea57ddbda748fb751"
server_key_version: "encryptionkey@0"
I20260926 02:13:55.117355 2081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:13:55.121114 2095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:55.122660 2081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:55.122815 2081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "025229d27ba44b958bac077d8833c568"
format_stamp: "Formatted at 2026-09-26 02:13:55 on dist-test-slave-hllj"
server_key: "23f4ba4fa568795ff365dafb6368365e"
server_key_iv: "af08085c7f69e65ea57ddbda748fb751"
server_key_version: "encryptionkey@0"
I20260926 02:13:55.123005 2081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:55.138702 2081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:55.139545 2081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:55.139820 2081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:55.157280 2081 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:38869
I20260926 02:13:55.157316 2146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:38869 every 8 connection(s)
I20260926 02:13:55.158644 2081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:55.162006 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 2081
I20260926 02:13:55.162230 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:55.163005 2147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:55.163024 29282 external_mini_cluster.cc:1428] Setting key 09de90658f425375d94ff0d149421c74
I20260926 02:13:55.176214 2147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568: Bootstrap starting.
I20260926 02:13:55.176996 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:55.165045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.28.152.190:38869)
0926 02:13:55.166155 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:55.166167 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:55.166213 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:13:55.168163 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:55.168201 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:55.168763 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:55.169205 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:55.169820 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:55.171122 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:55.172420 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:55.173181 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:55.173389 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:55.174542 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:55.174550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:55.174572 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:55.175127 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:55.175225 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:55.175246 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:55.175358 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:55.175579 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 02:13:55.175894 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":374,"threads_started":1}
I20260926 02:13:55.179478 2147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:55.180596 2147 log.cc:824] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:55.180794 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:13:55.183496 2147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568: No bootstrap required, opened a new log
I20260926 02:13:55.191385 2147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025229d27ba44b958bac077d8833c568" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38869 } }
I20260926 02:13:55.191764 2147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:55.191871 2147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025229d27ba44b958bac077d8833c568, State: Initialized, Role: FOLLOWER
I20260926 02:13:55.192451 2147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025229d27ba44b958bac077d8833c568" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38869 } }
I20260926 02:13:55.192668 2147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:55.192794 2147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:55.192951 2147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:55.195124 2147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025229d27ba44b958bac077d8833c568" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38869 } }
I20260926 02:13:55.195647 2147 leader_election.cc:302] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 025229d27ba44b958bac077d8833c568; no voters:
I20260926 02:13:55.196219 2147 leader_election.cc:288] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:55.196367 2152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:55.197014 2152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [term 1 LEADER]: Becoming Leader. State: Replica: 025229d27ba44b958bac077d8833c568, State: Running, Role: LEADER
I20260926 02:13:55.197615 2152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025229d27ba44b958bac077d8833c568" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38869 } }
I20260926 02:13:55.198391 2147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:55.199939 2154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025229d27ba44b958bac077d8833c568. Latest consensus state: current_term: 1 leader_uuid: "025229d27ba44b958bac077d8833c568" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025229d27ba44b958bac077d8833c568" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38869 } } }
I20260926 02:13:55.200273 2154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:55.201229 2159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:55.201339 2153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "025229d27ba44b958bac077d8833c568" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025229d27ba44b958bac077d8833c568" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38869 } } }
I20260926 02:13:55.201694 2153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:55.208045 2159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:55.219165 2159 catalog_manager.cc:1347] Generated new cluster ID: a0f92bcda7644ac099d75dac2e4fdc66
I20260926 02:13:55.219301 2159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:55.230756 2159 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:55.232689 2159 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:55.242549 2159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 025229d27ba44b958bac077d8833c568: Generated new TSK 0
I20260926 02:13:55.243798 2159 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 02:13:55.279809 2177 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 02:13:55.279927 2177 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:13:55.281324 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:55.260448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.28.152.190:38869)
0926 02:13:55.260667 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:55.260676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:55.261087 (+ 411us) server_negotiation.cc:407] Connection header received
0926 02:13:55.261374 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:55.261381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:55.261488 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:55.261704 (+ 216us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:13:55.262216 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:55.263371 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:55.264920 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:55.265683 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:55.265821 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:55.266367 (+ 546us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:13:55.280532 (+ 14165us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 02:13:55.280965 (+ 433us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59944: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 02:13:55.281431 2176 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:13:55.260303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:38869 (local address 127.0.0.1:59944)
0926 02:13:55.260965 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:55.260986 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:55.261125 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:55.261740 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:55.261744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:55.261756 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:13:55.262034 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:55.262045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:55.263599 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:55.263603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:55.264748 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:55.264763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:55.265860 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:55.265865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:55.266012 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:55.266053 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:13:55.280849 (+ 14796us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 02:13:55.281068 (+ 219us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.152.190:38869: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":373,"threads_started":1}
W20260926 02:13:55.281459 2150 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59944: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 02:13:55.282810 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 2081
I20260926 02:13:55.284000 2142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:55.375201 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 2081
2026-09-26T02:13:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (712 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 02:13:55.427227 29282 mini_oidc.cc:152] Starting JWKS server
I20260926 02:13:55.429090 29282 webserver.cc:460] Webserver started at https://127.0.0.1:34743/ using document root <none> and password file <none>
I20260926 02:13:55.429339 29282 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:13:55.430009 29282 webserver.cc:460] Webserver started at http://127.0.0.1:39887/ using document root <none> and password file <none>
2026-09-26T02:13:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:55Z Disabled control of system clock
I20260926 02:13:55.456161 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:42261
--webserver_interface=127.28.152.190
--webserver_port=0
--builtin_ntp_servers=127.28.152.148:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:42261
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34743/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 02:13:55.784107 2184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:55.784415 2184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:55.784664 2184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:55.792112 2184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:55.792236 2184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:55.792299 2184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:55.792376 2184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:55.792441 2184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:13:55.792503 2184 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:13:55.792557 2184 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34743/jwks.json
I20260926 02:13:55.801858 2184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:42261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:42261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34743/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:55.803201 2184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:55.805014 2184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:55.816738 2189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:55.816794 2190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:55.817117 2192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:55.817708 2184 server_base.cc:1034] running on GCE node
I20260926 02:13:55.818511 2184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:55.819949 2184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:55.821199 2184 hybrid_clock.cc:648] HybridClock initialized: now 1790388835821111 us; error 39 us; skew 500 ppm
I20260926 02:13:55.821602 2184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:13:55.824007 2184 webserver.cc:460] Webserver started at http://127.28.152.190:40275/ using document root <none> and password file <none>
I20260926 02:13:55.824775 2184 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:55.824882 2184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:55.825263 2184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:55.828253 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "605a363097bc414e9e4f84371b2c323d"
format_stamp: "Formatted at 2026-09-26 02:13:55 on dist-test-slave-hllj"
server_key: "1c774d9a6bb5d1d1bdf2706f2d9dfbae"
server_key_iv: "c223c40f8a322f52e0f9de2c4cdd2375"
server_key_version: "encryptionkey@0"
I20260926 02:13:55.829099 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "605a363097bc414e9e4f84371b2c323d"
format_stamp: "Formatted at 2026-09-26 02:13:55 on dist-test-slave-hllj"
server_key: "1c774d9a6bb5d1d1bdf2706f2d9dfbae"
server_key_iv: "c223c40f8a322f52e0f9de2c4cdd2375"
server_key_version: "encryptionkey@0"
I20260926 02:13:55.834594 2184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:13:55.838323 2198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:55.839875 2184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:13:55.840065 2184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "605a363097bc414e9e4f84371b2c323d"
format_stamp: "Formatted at 2026-09-26 02:13:55 on dist-test-slave-hllj"
server_key: "1c774d9a6bb5d1d1bdf2706f2d9dfbae"
server_key_iv: "c223c40f8a322f52e0f9de2c4cdd2375"
server_key_version: "encryptionkey@0"
I20260926 02:13:55.840301 2184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:55.853472 2184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:55.854337 2184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:55.854641 2184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:55.872088 2184 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:42261
I20260926 02:13:55.872094 2249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:42261 every 8 connection(s)
I20260926 02:13:55.873535 2184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:55.875622 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 2184
I20260926 02:13:55.875809 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:55.876454 29282 external_mini_cluster.cc:1428] Setting key 365d67b0419ffbfb97d85a4507b7d184
I20260926 02:13:55.877853 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:55.892263 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:55.878465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.28.152.190:42261)
0926 02:13:55.879588 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:55.879604 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:55.879656 (+ 52us) server_negotiation.cc:407] Connection header received
0926 02:13:55.881404 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:55.881437 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:55.882163 (+ 726us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:55.882653 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:55.883233 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:55.885342 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:55.886750 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:55.887763 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:55.887980 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:55.889158 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:55.889169 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:55.889191 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:55.889996 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:55.890107 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:55.890138 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:55.890262 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:55.890487 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 02:13:55.890884 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":364,"threads_started":1}
I20260926 02:13:55.892962 2250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d: Bootstrap starting.
I20260926 02:13:55.896314 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:13:55.896384 2250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:55.897706 2250 log.cc:824] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:55.900640 2250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d: No bootstrap required, opened a new log
I20260926 02:13:55.907932 2250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a363097bc414e9e4f84371b2c323d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42261 } }
I20260926 02:13:55.908360 2250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:55.908479 2250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605a363097bc414e9e4f84371b2c323d, State: Initialized, Role: FOLLOWER
I20260926 02:13:55.909173 2250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a363097bc414e9e4f84371b2c323d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42261 } }
I20260926 02:13:55.909407 2250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:55.909538 2250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:55.909699 2250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:55.911898 2250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a363097bc414e9e4f84371b2c323d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42261 } }
I20260926 02:13:55.912472 2250 leader_election.cc:302] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 605a363097bc414e9e4f84371b2c323d; no voters:
I20260926 02:13:55.913215 2250 leader_election.cc:288] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:55.913383 2255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:55.914101 2255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [term 1 LEADER]: Becoming Leader. State: Replica: 605a363097bc414e9e4f84371b2c323d, State: Running, Role: LEADER
I20260926 02:13:55.914789 2255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a363097bc414e9e4f84371b2c323d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42261 } }
I20260926 02:13:55.915613 2250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:55.918267 2256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "605a363097bc414e9e4f84371b2c323d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a363097bc414e9e4f84371b2c323d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42261 } } }
I20260926 02:13:55.918246 2257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 605a363097bc414e9e4f84371b2c323d. Latest consensus state: current_term: 1 leader_uuid: "605a363097bc414e9e4f84371b2c323d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605a363097bc414e9e4f84371b2c323d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42261 } } }
I20260926 02:13:55.918706 2256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:55.918706 2257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:55.919602 2263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:55.925273 2263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:55.934931 2263 catalog_manager.cc:1347] Generated new cluster ID: 06b21ff8cc6b472181f9304ec5c64003
I20260926 02:13:55.935111 2263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:55.958959 2263 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:55.961251 2263 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:55.978765 2263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 605a363097bc414e9e4f84371b2c323d: Generated new TSK 0
I20260926 02:13:55.979975 2263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:13:56.006901 2253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:55.997821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.28.152.190:42261)
0926 02:13:55.998014 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:55.998021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:55.998448 (+ 427us) server_negotiation.cc:407] Connection header received
0926 02:13:55.998682 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:55.998688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:55.998782 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:55.998951 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:13:55.999425 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:56.000553 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:56.002153 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:56.002899 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:56.003039 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:56.003677 (+ 638us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:13:56.006242 (+ 2565us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 02:13:56.006596 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35148: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 02:13:56.007040 2253 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35148: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 02:13:56.008451 2279 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:13:55.997673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:42261 (local address 127.0.0.1:35148)
0926 02:13:55.998328 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:55.998345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:55.998492 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:55.998996 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:55.999001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:55.999011 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:13:55.999262 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:55.999269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:56.000754 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:56.000759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:56.001984 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:56.002000 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:56.003177 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:56.003182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:56.003328 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:56.003366 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:13:56.006468 (+ 3102us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 02:13:56.008010 (+ 1542us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.152.190:42261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":1239,"thread_start_us":358,"threads_started":1}
I20260926 02:13:56.009701 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 2184
I20260926 02:13:56.010965 2245 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 02:13:56.018057 2280 webserver.cc:533] Webserver: sslize failed:
E20260926 02:13:56.018159 2280 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:13:56.102519 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 2184
2026-09-26T02:13:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (727 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T02:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:56Z Disabled control of system clock
I20260926 02:13:56.180946 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:46007
--webserver_interface=127.28.152.190
--webserver_port=0
--builtin_ntp_servers=127.28.152.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:46007
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/testchainca.cert with env {}
W20260926 02:13:56.509054 2285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:13:56.509383 2285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:13:56.509511 2285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:13:56.516793 2285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:13:56.516927 2285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:13:56.516968 2285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:13:56.517041 2285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:13:56.517098 2285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:13:56.517199 2285 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:13:56.517246 2285 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 02:13:56.527755 2285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:46007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:46007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:13:56.529281 2285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:13:56.530637 2285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:13:56.542348 2293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:56.542408 2291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:13:56.542455 2290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:13:56.543108 2285 server_base.cc:1034] running on GCE node
I20260926 02:13:56.543963 2285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:13:56.545516 2285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:13:56.546737 2285 hybrid_clock.cc:648] HybridClock initialized: now 1790388836546690 us; error 40 us; skew 500 ppm
I20260926 02:13:56.547101 2285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:13:56.549422 2285 webserver.cc:460] Webserver started at http://127.28.152.190:40165/ using document root <none> and password file <none>
I20260926 02:13:56.550151 2285 fs_manager.cc:359] Metadata directory not provided
I20260926 02:13:56.550262 2285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:13:56.550578 2285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:13:56.553676 2285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "ad009b2ad7ce499791ae8ee6e2b4b842"
format_stamp: "Formatted at 2026-09-26 02:13:56 on dist-test-slave-hllj"
server_key: "e02cb93dd5ca82d75822c36d6daeaad6"
server_key_iv: "4b8b7b057648b47a6eaa7828ecca8458"
server_key_version: "encryptionkey@0"
I20260926 02:13:56.554490 2285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "ad009b2ad7ce499791ae8ee6e2b4b842"
format_stamp: "Formatted at 2026-09-26 02:13:56 on dist-test-slave-hllj"
server_key: "e02cb93dd5ca82d75822c36d6daeaad6"
server_key_iv: "4b8b7b057648b47a6eaa7828ecca8458"
server_key_version: "encryptionkey@0"
I20260926 02:13:56.559760 2285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:13:56.563248 2299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:13:56.564703 2285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:13:56.564867 2285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "ad009b2ad7ce499791ae8ee6e2b4b842"
format_stamp: "Formatted at 2026-09-26 02:13:56 on dist-test-slave-hllj"
server_key: "e02cb93dd5ca82d75822c36d6daeaad6"
server_key_iv: "4b8b7b057648b47a6eaa7828ecca8458"
server_key_version: "encryptionkey@0"
I20260926 02:13:56.565059 2285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:13:56.582273 2285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:13:56.583189 2285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:13:56.583465 2285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:13:56.601119 2285 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:46007
I20260926 02:13:56.601158 2350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:46007 every 8 connection(s)
I20260926 02:13:56.602576 2285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:13:56.606496 2351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:13:56.610266 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 2285
I20260926 02:13:56.610457 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:13:56.611047 29282 external_mini_cluster.cc:1428] Setting key ca069317ffe0a8fd7208e947478480fc
I20260926 02:13:56.618940 2351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842: Bootstrap starting.
I20260926 02:13:56.622738 2351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842: Neither blocks nor log segments found. Creating new log.
I20260926 02:13:56.624066 2351 log.cc:824] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842: Log is configured to *not* fsync() on all Append() calls
I20260926 02:13:56.627560 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:56.612989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.28.152.190:46007)
0926 02:13:56.614146 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:56.614163 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:56.614215 (+ 52us) server_negotiation.cc:407] Connection header received
0926 02:13:56.616978 (+ 2763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:56.617013 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:56.617762 (+ 749us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:56.618291 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:13:56.618941 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:56.620909 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:56.622306 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:56.623388 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:56.623565 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:56.624724 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:13:56.624734 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:13:56.624759 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:13:56.625391 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:13:56.625495 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:13:56.625529 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:13:56.625637 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:13:56.625896 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 02:13:56.626261 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":360,"threads_started":1}
I20260926 02:13:56.627698 2351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842: No bootstrap required, opened a new log
I20260926 02:13:56.631898 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:13:56.637293 2351 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad009b2ad7ce499791ae8ee6e2b4b842" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46007 } }
I20260926 02:13:56.637671 2351 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:13:56.637742 2351 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad009b2ad7ce499791ae8ee6e2b4b842, State: Initialized, Role: FOLLOWER
I20260926 02:13:56.638240 2351 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad009b2ad7ce499791ae8ee6e2b4b842" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46007 } }
I20260926 02:13:56.638434 2351 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:13:56.638550 2351 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:13:56.638698 2351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:13:56.640902 2351 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad009b2ad7ce499791ae8ee6e2b4b842" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46007 } }
I20260926 02:13:56.641405 2351 leader_election.cc:302] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad009b2ad7ce499791ae8ee6e2b4b842; no voters:
I20260926 02:13:56.641953 2351 leader_election.cc:288] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:13:56.642087 2356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:13:56.642911 2356 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [term 1 LEADER]: Becoming Leader. State: Replica: ad009b2ad7ce499791ae8ee6e2b4b842, State: Running, Role: LEADER
I20260926 02:13:56.643489 2356 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad009b2ad7ce499791ae8ee6e2b4b842" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46007 } }
I20260926 02:13:56.643991 2351 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:13:56.646296 2358 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad009b2ad7ce499791ae8ee6e2b4b842. Latest consensus state: current_term: 1 leader_uuid: "ad009b2ad7ce499791ae8ee6e2b4b842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad009b2ad7ce499791ae8ee6e2b4b842" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46007 } } }
I20260926 02:13:56.646358 2357 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad009b2ad7ce499791ae8ee6e2b4b842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad009b2ad7ce499791ae8ee6e2b4b842" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46007 } } }
I20260926 02:13:56.646651 2358 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:56.646692 2357 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842 [sys.catalog]: This master's current role is: LEADER
I20260926 02:13:56.647509 2362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:13:56.653450 2362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:13:56.663122 2362 catalog_manager.cc:1347] Generated new cluster ID: c19d3d89c08e4cf2a0e170c01a45c5f0
I20260926 02:13:56.663244 2362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:13:56.684872 2362 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:13:56.686867 2362 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:13:56.697057 2362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad009b2ad7ce499791ae8ee6e2b4b842: Generated new TSK 0
I20260926 02:13:56.698233 2362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:13:56.818379 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:13:56.710717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33514 (local address 127.28.152.190:46007)
0926 02:13:56.710874 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 02:13:56.710880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:13:56.711337 (+ 457us) server_negotiation.cc:407] Connection header received
0926 02:13:56.711555 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:13:56.711558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:13:56.711640 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:13:56.711774 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:13:56.712220 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:56.712971 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:56.714407 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:13:56.714859 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:13:56.714974 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:56.715491 (+ 517us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:13:56.817565 (+102074us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 02:13:56.818040 (+ 475us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33514: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 02:13:56.818548 2354 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33514: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 02:13:56.818776 2380 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:13:56.710665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:46007 (local address 127.0.0.1:33514)
0926 02:13:56.711231 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 02:13:56.711246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:13:56.711391 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:13:56.711808 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:13:56.711812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:13:56.711824 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:13:56.712081 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:56.712090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:56.713116 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:56.713120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:56.714267 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:13:56.714279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:13:56.715032 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:13:56.715039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:13:56.715170 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:13:56.715204 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:13:56.817888 (+102684us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 02:13:56.818230 (+ 342us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.152.190:46007: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":308,"threads_started":1}
I20260926 02:13:56.819852 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 2285
I20260926 02:13:56.821118 2346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:13:56.922972 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 2285
2026-09-26T02:13:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (800 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (405 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1381 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:13:58 dist-test-slave-hllj krb5kdc[2396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:13:58 dist-test-slave-hllj krb5kdc[2396](info): set up 2 sockets
Sep 26 02:13:58 dist-test-slave-hllj krb5kdc[2396](info): commencing operation
krb5kdc: starting...
W20260926 02:13:59.731163 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:13:59 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:13:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:13:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:13:59.938755 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:39473
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:00.263728 2409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:00.264034 2409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:00.264122 2409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:00.271275 2409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:14:00.271390 2409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:00.271436 2409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:00.271492 2409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:14:00.271548 2409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:14:00.281383 2409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:39473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:00.282691 2409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:00.284008 2409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:00.295812 2414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:00.295769 2415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:00.295790 2417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:00.296543 2409 server_base.cc:1034] running on GCE node
I20260926 02:14:00.297401 2409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:00.298852 2409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:00.300100 2409 hybrid_clock.cc:648] HybridClock initialized: now 1790388840300040 us; error 49 us; skew 500 ppm
I20260926 02:14:00.300460 2409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:00 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:00.303665 2409 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:14:00.305163 2409 webserver.cc:460] Webserver started at http://127.28.152.190:39527/ using document root <none> and password file <none>
I20260926 02:14:00.305871 2409 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:00.305992 2409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:00.306270 2409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:00.309427 2409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "2a28f1f86c1b4e6b9599ded9aa707fe3"
format_stamp: "Formatted at 2026-09-26 02:14:00 on dist-test-slave-hllj"
server_key: "d55838a51116493429b81a377fdce7d6"
server_key_iv: "58302e9805c0e4bea2d01be4dc28e2a3"
server_key_version: "encryptionkey@0"
I20260926 02:14:00.310238 2409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "2a28f1f86c1b4e6b9599ded9aa707fe3"
format_stamp: "Formatted at 2026-09-26 02:14:00 on dist-test-slave-hllj"
server_key: "d55838a51116493429b81a377fdce7d6"
server_key_iv: "58302e9805c0e4bea2d01be4dc28e2a3"
server_key_version: "encryptionkey@0"
I20260926 02:14:00.315644 2409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 02:14:00.319651 2424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:00.321434 2409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:14:00.321640 2409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "2a28f1f86c1b4e6b9599ded9aa707fe3"
format_stamp: "Formatted at 2026-09-26 02:14:00 on dist-test-slave-hllj"
server_key: "d55838a51116493429b81a377fdce7d6"
server_key_iv: "58302e9805c0e4bea2d01be4dc28e2a3"
server_key_version: "encryptionkey@0"
I20260926 02:14:00.321846 2409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:00.335357 2409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:00.339965 2409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:00.340246 2409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:00.357570 2409 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:39473
I20260926 02:14:00.357627 2475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:39473 every 8 connection(s)
I20260926 02:14:00.359050 2409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:14:00.363168 2476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:00.369240 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 2409
I20260926 02:14:00.369503 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:14:00.370273 29282 external_mini_cluster.cc:1428] Setting key ff72128f3b3c631e0392301d55f6cdfc
I20260926 02:14:00.375329 2476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3: Bootstrap starting.
I20260926 02:14:00.378659 2476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:00.379773 2476 log.cc:824] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:00.383394 2476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3: No bootstrap required, opened a new log
Sep 26 02:14:00 dist-test-slave-hllj krb5kdc[2396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:00.393165 2476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f1f86c1b4e6b9599ded9aa707fe3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39473 } }
I20260926 02:14:00.393682 2476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:00.393812 2476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a28f1f86c1b4e6b9599ded9aa707fe3, State: Initialized, Role: FOLLOWER
I20260926 02:14:00.394546 2476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f1f86c1b4e6b9599ded9aa707fe3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39473 } }
I20260926 02:14:00.394860 2476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:14:00.395015 2476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:14:00.395201 2476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:00.395835 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:00.372692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.28.152.190:39473)
0926 02:14:00.373601 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:00.373615 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:00.373666 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:14:00.375525 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:00.375552 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:00.376011 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:00.376413 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:00.377937 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:00.379388 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:00.380978 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:00.382076 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:00.382295 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:00.386234 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:00.386247 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:00.386272 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:00.386331 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:00.390638 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:00.391532 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:00.391542 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:00.391553 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:00.391816 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:00.392229 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:00.392234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:00.392238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:00.393027 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:00.393358 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:00.393770 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 02:14:00.394282 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":435,"threads_started":1}
I20260926 02:14:00.398253 2476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f1f86c1b4e6b9599ded9aa707fe3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39473 } }
I20260926 02:14:00.398826 2476 leader_election.cc:302] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a28f1f86c1b4e6b9599ded9aa707fe3; no voters:
I20260926 02:14:00.399508 2476 leader_election.cc:288] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:14:00.399653 2481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:00.400358 2481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [term 1 LEADER]: Becoming Leader. State: Replica: 2a28f1f86c1b4e6b9599ded9aa707fe3, State: Running, Role: LEADER
I20260926 02:14:00.400926 2481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f1f86c1b4e6b9599ded9aa707fe3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39473 } }
I20260926 02:14:00.401675 2476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:14:00.403868 2482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a28f1f86c1b4e6b9599ded9aa707fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f1f86c1b4e6b9599ded9aa707fe3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39473 } } }
I20260926 02:14:00.404114 2483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a28f1f86c1b4e6b9599ded9aa707fe3. Latest consensus state: current_term: 1 leader_uuid: "2a28f1f86c1b4e6b9599ded9aa707fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a28f1f86c1b4e6b9599ded9aa707fe3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39473 } } }
I20260926 02:14:00.404537 2483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:00.404537 2482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:00.405719 2488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:14:00.411720 2488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:14:00.422562 2488 catalog_manager.cc:1347] Generated new cluster ID: 9c545f83fc8b4d8c8b28e1dcfc7747ee
I20260926 02:14:00.422691 2488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:14:00.436632 2488 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:14:00.438755 2488 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:14:00.452836 2488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a28f1f86c1b4e6b9599ded9aa707fe3: Generated new TSK 0
I20260926 02:14:00.454134 2488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:14:00.512562 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39473
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:00.842499 2502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:00.842808 2502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:00.842943 2502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:00.850296 2502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:00.850419 2502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:00.850567 2502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:14:00.860345 2502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:00.861742 2502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:00.863199 2502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:00.876591 2507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:00.876652 2510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:00.876652 2508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:00.877930 2502 server_base.cc:1034] running on GCE node
I20260926 02:14:00.878584 2502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:00.879622 2502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:00.880851 2502 hybrid_clock.cc:648] HybridClock initialized: now 1790388840880781 us; error 68 us; skew 500 ppm
I20260926 02:14:00.881244 2502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:00 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:00.884438 2502 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:14:00.886247 2502 webserver.cc:460] Webserver started at http://127.28.152.129:40801/ using document root <none> and password file <none>
I20260926 02:14:00.886967 2502 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:00.887099 2502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:00.887408 2502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:00.890503 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "c8d93ce09aed459cb6f174709b3f7a83"
format_stamp: "Formatted at 2026-09-26 02:14:00 on dist-test-slave-hllj"
server_key: "c148f8c7f449a232ef1d85c110ce0e4e"
server_key_iv: "87c2e7f4ac58f6fdcb81cdc34c759beb"
server_key_version: "encryptionkey@0"
I20260926 02:14:00.891328 2502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "c8d93ce09aed459cb6f174709b3f7a83"
format_stamp: "Formatted at 2026-09-26 02:14:00 on dist-test-slave-hllj"
server_key: "c148f8c7f449a232ef1d85c110ce0e4e"
server_key_iv: "87c2e7f4ac58f6fdcb81cdc34c759beb"
server_key_version: "encryptionkey@0"
I20260926 02:14:00.897076 2502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 02:14:00.902197 2517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:00.904222 2502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:14:00.904417 2502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "c8d93ce09aed459cb6f174709b3f7a83"
format_stamp: "Formatted at 2026-09-26 02:14:00 on dist-test-slave-hllj"
server_key: "c148f8c7f449a232ef1d85c110ce0e4e"
server_key_iv: "87c2e7f4ac58f6fdcb81cdc34c759beb"
server_key_version: "encryptionkey@0"
I20260926 02:14:00.904664 2502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:00.941171 2502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:00.945537 2502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:00.945819 2502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:00.946794 2502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:00.948130 2502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:00.948210 2502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:00.948338 2502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:00.948391 2502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:00.978464 2502 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:36999
I20260926 02:14:00.978515 2629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:36999 every 8 connection(s)
I20260926 02:14:00.979759 2502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:14:00.982997 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 2502
I20260926 02:14:00.983246 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:14:00.984208 29282 external_mini_cluster.cc:1428] Setting key eb62d2edde638818c537afeb3ae42464
Sep 26 02:14:00 dist-test-slave-hllj krb5kdc[2396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:01.000519 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:00.982297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:37143 (local address 127.28.152.190:39473)
0926 02:14:00.982728 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:00.982732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:00.983947 (+ 1215us) server_negotiation.cc:407] Connection header received
0926 02:14:00.985919 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:00.985924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:00.986020 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:00.986109 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:00.987757 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:00.988480 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:00.990172 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:00.990578 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:00.990705 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:00.994737 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:00.994743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:00.994747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:00.994782 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:00.997372 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:00.998152 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:00.998157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:00.998159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:00.998282 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:00.998775 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:00.998779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:00.998781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:00.999050 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:00.999180 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:00.999939 (+ 759us) server_negotiation.cc:299] Negotiation successful
0926 02:14:01.000169 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":216,"threads_started":1}
I20260926 02:14:01.001670 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:00.982957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39473 (local address 127.28.152.129:37143)
0926 02:14:00.983771 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:00.983819 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:00.985633 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:00.986393 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:00.986402 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:00.986938 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:00.987569 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:00.987589 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:00.988636 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:00.988640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:00.990013 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:00.990026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:00.990832 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:00.990839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:00.991104 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:00.991860 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:00.991895 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:00.994481 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:00.997528 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:00.997539 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:00.997561 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:00.998010 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:00.998390 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:00.998395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:00.998398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:00.998653 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:00.999173 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:00.999181 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:00.999553 (+ 372us) client_negotiation.cc:769] Sending connection context
0926 02:14:00.999801 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 02:14:01.000376 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
I20260926 02:14:01.003317 2630 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39473
I20260926 02:14:01.003609 2630 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:01.004364 2630 heartbeater.cc:511] Master 127.28.152.190:39473 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
I20260926 02:14:01.006618 2441 ts_manager.cc:194] Registered new tserver with Master: c8d93ce09aed459cb6f174709b3f7a83 (127.28.152.129:36999)
I20260926 02:14:01.008061 2441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:37143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:14:01.028995 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39473
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:01.360423 2636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:01.360816 2636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:01.360961 2636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:01.372800 2636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:01.372963 2636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:01.373175 2636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:14:01.385675 2636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:01.386996 2636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:01.388335 2636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:01.401995 2642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:01.401995 2641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:01.401980 2644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:01.403327 2636 server_base.cc:1034] running on GCE node
I20260926 02:14:01.403987 2636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:01.404973 2636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:01.406196 2636 hybrid_clock.cc:648] HybridClock initialized: now 1790388841406119 us; error 82 us; skew 500 ppm
I20260926 02:14:01.406531 2636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:01 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:01.410030 2636 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:14:01.411839 2636 webserver.cc:460] Webserver started at http://127.28.152.130:46671/ using document root <none> and password file <none>
I20260926 02:14:01.412561 2636 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:01.412680 2636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:01.412997 2636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:01.416100 2636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "14a6cdbacee54aa6a91a5793ae3f9df2"
format_stamp: "Formatted at 2026-09-26 02:14:01 on dist-test-slave-hllj"
server_key: "8b2b9fbee684b09eba2a56edf52bc5e5"
server_key_iv: "ae0aa529e8807ad161c8481ae78dc832"
server_key_version: "encryptionkey@0"
I20260926 02:14:01.416923 2636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "14a6cdbacee54aa6a91a5793ae3f9df2"
format_stamp: "Formatted at 2026-09-26 02:14:01 on dist-test-slave-hllj"
server_key: "8b2b9fbee684b09eba2a56edf52bc5e5"
server_key_iv: "ae0aa529e8807ad161c8481ae78dc832"
server_key_version: "encryptionkey@0"
I20260926 02:14:01.422537 2636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:14:01.426388 2651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:01.428033 2636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:14:01.428206 2636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "14a6cdbacee54aa6a91a5793ae3f9df2"
format_stamp: "Formatted at 2026-09-26 02:14:01 on dist-test-slave-hllj"
server_key: "8b2b9fbee684b09eba2a56edf52bc5e5"
server_key_iv: "ae0aa529e8807ad161c8481ae78dc832"
server_key_version: "encryptionkey@0"
I20260926 02:14:01.428406 2636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:01.449862 2636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:01.454248 2636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:01.454528 2636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:01.455500 2636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:01.456846 2636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:01.456920 2636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:01.457026 2636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:01.457082 2636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:01.486440 2636 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:37459
I20260926 02:14:01.486487 2763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:37459 every 8 connection(s)
I20260926 02:14:01.487761 2636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:14:01.489828 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 2636
I20260926 02:14:01.490006 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:14:01.490448 29282 external_mini_cluster.cc:1428] Setting key a101b594ccae9ab490007cc7df01efcf
Sep 26 02:14:01 dist-test-slave-hllj krb5kdc[2396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:01.510886 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:01.490412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:56943 (local address 127.28.152.190:39473)
0926 02:14:01.490613 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:01.490619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:01.492000 (+ 1381us) server_negotiation.cc:407] Connection header received
0926 02:14:01.494919 (+ 2919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:01.494924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:01.494994 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:01.495071 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:01.497052 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:01.497803 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:01.499837 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:01.500466 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:01.500606 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:01.504365 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:01.504371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:01.504374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:01.504411 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:01.507010 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:01.508036 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:01.508041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:01.508043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:01.508199 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:01.508751 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:01.508755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:01.508758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:01.509155 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:01.509293 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:01.510357 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0926 02:14:01.510633 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:14:01.512264 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:01.490948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39473 (local address 127.28.152.130:56943)
0926 02:14:01.491826 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:01.491881 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:01.494565 (+ 2684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:01.495355 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:01.495367 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:01.496066 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:01.496875 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:01.496904 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:01.497954 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:01.497959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:01.499701 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:01.499715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:01.500684 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:01.500694 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:01.500902 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:01.501602 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:01.501625 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:01.504095 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:01.507197 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:01.507208 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:01.507231 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:01.507882 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:01.508316 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:01.508322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:01.508326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:01.508598 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:01.509296 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:01.509306 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:01.509810 (+ 504us) client_negotiation.cc:769] Sending connection context
0926 02:14:01.510240 (+ 430us) client_negotiation.cc:240] Negotiation successful
0926 02:14:01.510851 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":354,"threads_started":1}
I20260926 02:14:01.513734 2764 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
I20260926 02:14:01.514086 2764 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:01.514992 2764 heartbeater.cc:511] Master 127.28.152.190:39473 requested a full tablet report, sending...
I20260926 02:14:01.516754 2441 ts_manager.cc:194] Registered new tserver with Master: 14a6cdbacee54aa6a91a5793ae3f9df2 (127.28.152.130:37459)
I20260926 02:14:01.517645 2441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:56943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:14:01.538749 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39473
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:01.864706 2769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:01.865027 2769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:01.865170 2769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:01.872534 2769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:01.872682 2769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:01.872838 2769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:14:01.882908 2769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:01.884239 2769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:01.885604 2769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:01.898891 2774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:01.898770 2775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:01.898772 2777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:01.900041 2769 server_base.cc:1034] running on GCE node
I20260926 02:14:01.900718 2769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:01.901830 2769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:01.903054 2769 hybrid_clock.cc:648] HybridClock initialized: now 1790388841902997 us; error 43 us; skew 500 ppm
I20260926 02:14:01.903386 2769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:01 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:01.906689 2769 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:14:01.908429 2769 webserver.cc:460] Webserver started at http://127.28.152.131:34709/ using document root <none> and password file <none>
I20260926 02:14:01.909312 2769 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:01.909462 2769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:01.909781 2769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:01.912865 2769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "3412ff97bed3496ca45c4e6b7ccabace"
format_stamp: "Formatted at 2026-09-26 02:14:01 on dist-test-slave-hllj"
server_key: "29d42433183a1fc3a77c33725240247f"
server_key_iv: "50388d522561b70c0e2bfaa2cee9b50f"
server_key_version: "encryptionkey@0"
I20260926 02:14:01.913857 2769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "3412ff97bed3496ca45c4e6b7ccabace"
format_stamp: "Formatted at 2026-09-26 02:14:01 on dist-test-slave-hllj"
server_key: "29d42433183a1fc3a77c33725240247f"
server_key_iv: "50388d522561b70c0e2bfaa2cee9b50f"
server_key_version: "encryptionkey@0"
I20260926 02:14:01.919698 2769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 02:14:01.923540 2784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:01.925035 2769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:14:01.925308 2769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "3412ff97bed3496ca45c4e6b7ccabace"
format_stamp: "Formatted at 2026-09-26 02:14:01 on dist-test-slave-hllj"
server_key: "29d42433183a1fc3a77c33725240247f"
server_key_iv: "50388d522561b70c0e2bfaa2cee9b50f"
server_key_version: "encryptionkey@0"
I20260926 02:14:01.925580 2769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:01.947347 2769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:01.952036 2769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:01.952334 2769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:01.953384 2769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:01.954764 2769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:01.954847 2769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:01.954969 2769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:01.955020 2769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:01.985415 2769 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:32883
I20260926 02:14:01.985445 2896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:32883 every 8 connection(s)
I20260926 02:14:01.986735 2769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:14:01.988713 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 2769
I20260926 02:14:01.988900 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:14:01.989472 29282 external_mini_cluster.cc:1428] Setting key 03fe0e19321035e98d561958786a0e55
Sep 26 02:14:01 dist-test-slave-hllj krb5kdc[2396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:02.006959 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:01.989383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:56443 (local address 127.28.152.190:39473)
0926 02:14:01.989560 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:01.989566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:01.990964 (+ 1398us) server_negotiation.cc:407] Connection header received
0926 02:14:01.993151 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:01.993157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:01.993233 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:01.993336 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:01.994893 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:01.995655 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:01.997354 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:01.997755 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:01.997850 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:02.001229 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:02.001235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:02.001238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:02.001273 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:02.003817 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.004594 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.004598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.004601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.004735 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.005263 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.005267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.005271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.005610 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:02.005797 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:02.006527 (+ 730us) server_negotiation.cc:299] Negotiation successful
0926 02:14:02.006715 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:14:02.007956 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:01.989980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39473 (local address 127.28.152.131:56443)
0926 02:14:01.990825 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:01.990869 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:01.992828 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:01.993621 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:01.993630 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:01.994147 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:01.994734 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:01.994752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:01.995825 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:01.995829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:01.997202 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:01.997216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:01.997948 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:01.997953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:01.998214 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:01.998816 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:01.998843 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:02.000957 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:02.003957 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:02.003967 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:02.003986 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:02.004471 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:02.004839 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:02.004843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:02.004846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:02.005159 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:02.005742 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:02.005748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:02.006157 (+ 409us) client_negotiation.cc:769] Sending connection context
0926 02:14:02.006406 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 02:14:02.006832 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":311,"threads_started":1}
I20260926 02:14:02.009467 2897 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39473
I20260926 02:14:02.009783 2897 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:02.010582 2897 heartbeater.cc:511] Master 127.28.152.190:39473 requested a full tablet report, sending...
I20260926 02:14:02.011755 2630 heartbeater.cc:503] Master 127.28.152.190:39473 was elected leader, sending a full tablet report...
I20260926 02:14:02.012061 2441 ts_manager.cc:194] Registered new tserver with Master: 3412ff97bed3496ca45c4e6b7ccabace (127.28.152.131:32883)
I20260926 02:14:02.013121 2441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:56443
I20260926 02:14:02.018488 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/test-admin.keytab.
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:02.042920 29282 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 02:14:02.047791 2904 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:02.045486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.28.152.129:36999)
0926 02:14:02.046137 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:02.046144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:02.046175 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:14:02.046272 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:02.046279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:02.046505 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:02.046634 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:02.047415 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33416: BlockingRecv error: recv got EOF from 127.0.0.1:33416 (error 108)
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":312,"threads_started":1}
W20260926 02:14:02.048185 29282 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/krb5cc' of type FILE
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:02.049750 29282 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:02.051246 29282 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:14:02.066635 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:02.053030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.28.152.129:36999)
0926 02:14:02.053250 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:02.053256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:02.053331 (+ 75us) server_negotiation.cc:407] Connection header received
0926 02:14:02.053525 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:02.053529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:02.053596 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:02.053673 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:02.054700 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.055683 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.056940 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.057570 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.057706 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:02.061337 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:02.061351 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:02.061368 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:02.061405 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:02.064006 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.064727 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.064737 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.064741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.064895 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.065289 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.065292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.065294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.065653 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:02.065869 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:02.066149 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 02:14:02.066378 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 02:14:02.072512 2904 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:02.070768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33442 (local address 127.28.152.129:36999)
0926 02:14:02.070964 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:02.070970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:02.071108 (+ 138us) server_negotiation.cc:407] Connection header received
0926 02:14:02.071279 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:02.071284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:02.071343 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:02.071419 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:02.072265 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33442: BlockingRecv error: recv got EOF from 127.0.0.1:33442 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 02:14:02.072685 29282 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/krb5cc' of type FILE
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:02.074239 29282 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:02.075476 29282 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:14:02.090535 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:02.076815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.28.152.129:36999)
0926 02:14:02.076991 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:02.076997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:02.077317 (+ 320us) server_negotiation.cc:407] Connection header received
0926 02:14:02.077494 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:02.077497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:02.077553 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:02.077631 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:02.078572 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.079286 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.080535 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.080970 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.081097 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:02.084915 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:02.084921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:02.084924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:02.084964 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:02.087730 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.088513 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.088518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.088520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.088660 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.089060 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.089064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.089066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.089411 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:02.089667 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:02.090006 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 02:14:02.090249 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 02:14:02.095130 2904 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:02.093490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33472 (local address 127.28.152.129:36999)
0926 02:14:02.093688 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:02.093694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:02.093911 (+ 217us) server_negotiation.cc:407] Connection header received
0926 02:14:02.094047 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:02.094050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:02.094123 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:02.094215 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:02.094901 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33472: BlockingRecv error: recv got EOF from 127.0.0.1:33472 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 02:14:02.095321 29282 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790388791636863-29282-0/krb5kdc/krb5cc' of type FILE
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:02.096818 29282 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:02.098134 29282 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:14:02.112524 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:02.099473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.28.152.129:36999)
0926 02:14:02.099701 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:02.099707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:02.099934 (+ 227us) server_negotiation.cc:407] Connection header received
0926 02:14:02.100108 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:02.100112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:02.100175 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:02.100257 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:02.101214 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.101934 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.103305 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.103807 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.103916 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:02.107317 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:02.107323 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:02.107326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:02.107362 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:02.109963 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.110687 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.110692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.110694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.110856 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.111310 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.111313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.111315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.111597 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:02.111845 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:02.112029 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 02:14:02.112196 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:14:02.114217 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 2502
I20260926 02:14:02.125623 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:02.114964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.28.152.129:36999)
0926 02:14:02.115141 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:02.115147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:02.115400 (+ 253us) server_negotiation.cc:407] Connection header received
0926 02:14:02.115540 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:02.115544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:02.115605 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:02.115671 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:02.116515 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.117307 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.118610 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.119027 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.119130 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:02.121082 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:02.121088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:02.121091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:02.121120 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:02.123379 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.124005 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.124009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.124011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.124148 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.124422 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.124425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.124427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.124665 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:02.124853 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:02.125092 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 02:14:02.125289 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:14:02.126250 2625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:02.208071 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 2502
I20260926 02:14:02.220950 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 2636
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:14:02.236431 2917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:02.221889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.28.152.130:37459)
0926 02:14:02.222495 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:02.222512 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:02.222543 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:14:02.222620 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:02.222630 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:02.222890 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:02.223040 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:02.224054 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.225092 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.226357 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.226922 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.227033 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:02.230850 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:02.230867 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:02.230888 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:02.230931 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:02.233771 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.234438 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.234446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.234448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.234573 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.234912 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.234915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.234917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.235269 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:02.235548 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:02.235882 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 02:14:02.236139 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":270,"threads_started":1}
I20260926 02:14:02.237691 2759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:02.319113 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 2636
I20260926 02:14:02.335191 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 2769
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:14:02.350950 2919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:02.336037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.28.152.131:32883)
0926 02:14:02.336613 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:02.336620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:02.336648 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:14:02.336723 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:02.336729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:02.336934 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:02.337067 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:02.338451 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.339598 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.340967 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:02.341644 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:02.341754 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:02.345598 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:02.345610 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:02.345626 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:02.345663 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:02.348337 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.348986 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.348992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.348994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.349107 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:02.349526 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:02.349529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:02.349531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:02.349880 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:02.350092 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:02.350453 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 02:14:02.350667 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":273,"threads_started":1}
I20260926 02:14:02.352207 2892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:02.433326 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 2769
I20260926 02:14:02.445964 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 2409
I20260926 02:14:02.447223 2471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:02.547817 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 2409
2026-09-26T02:14:02Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3841 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2924](info): set up 2 sockets
Sep 26 02:14:02 dist-test-slave-hllj krb5kdc[2924](info): commencing operation
krb5kdc: starting...
W20260926 02:14:03.559053 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:14:03 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:14:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.190.keytab.
Entry for principal oryx/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.190.keytab.
I20260926 02:14:03.764734 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34891
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.190.keytab
--principal=oryx/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:04.093354 2937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:04.093689 2937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:04.093827 2937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:04.101289 2937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:14:04.101428 2937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:04.101507 2937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:04.101580 2937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:14:04.101642 2937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:14:04.112252 2937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.190.keytab
--principal=oryx/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:04.113746 2937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:04.115080 2937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:04.127345 2942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:04.127476 2943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:04.127795 2945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:04.128422 2937 server_base.cc:1034] running on GCE node
I20260926 02:14:04.129313 2937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:04.130810 2937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:04.132020 2937 hybrid_clock.cc:648] HybridClock initialized: now 1790388844131945 us; error 69 us; skew 500 ppm
I20260926 02:14:04.132364 2937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:04 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388844, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:04.142455 2937 init.cc:377] Logged in from keytab as oryx/127.28.152.190@KRBTEST.COM (short username oryx)
I20260926 02:14:04.144143 2937 webserver.cc:460] Webserver started at http://127.28.152.190:46073/ using document root <none> and password file <none>
I20260926 02:14:04.144860 2937 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:04.144984 2937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:04.145349 2937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:04.148461 2937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "248a9ebcadfd4363957532d4e5a993ec"
format_stamp: "Formatted at 2026-09-26 02:14:04 on dist-test-slave-hllj"
server_key: "9d14bfc28b62b723afc80ddbfcf48bbd"
server_key_iv: "17cacd3a487bd2cea128ffbc4ca7bd67"
server_key_version: "encryptionkey@0"
I20260926 02:14:04.149281 2937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "248a9ebcadfd4363957532d4e5a993ec"
format_stamp: "Formatted at 2026-09-26 02:14:04 on dist-test-slave-hllj"
server_key: "9d14bfc28b62b723afc80ddbfcf48bbd"
server_key_iv: "17cacd3a487bd2cea128ffbc4ca7bd67"
server_key_version: "encryptionkey@0"
I20260926 02:14:04.154796 2937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 02:14:04.158617 2952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:04.160110 2937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:14:04.160296 2937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "248a9ebcadfd4363957532d4e5a993ec"
format_stamp: "Formatted at 2026-09-26 02:14:04 on dist-test-slave-hllj"
server_key: "9d14bfc28b62b723afc80ddbfcf48bbd"
server_key_iv: "17cacd3a487bd2cea128ffbc4ca7bd67"
server_key_version: "encryptionkey@0"
I20260926 02:14:04.160495 2937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:04.181977 2937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:04.186080 2937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:04.186350 2937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:04.203648 2937 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:34891
I20260926 02:14:04.203665 3003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:34891 every 8 connection(s)
I20260926 02:14:04.205120 2937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:14:04.206691 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 2937
I20260926 02:14:04.206929 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:14:04.207728 29282 external_mini_cluster.cc:1428] Setting key b73e95e8a1489d0985e227f1d6dea197
I20260926 02:14:04.210299 3004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:14:04 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:04.223292 3004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec: Bootstrap starting.
I20260926 02:14:04.227507 3004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:04.228782 3004 log.cc:824] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:04.232206 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:04.209973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.28.152.190:34891)
0926 02:14:04.210726 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:04.210738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:04.210780 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:14:04.212382 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:04.212404 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:04.212830 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:04.213243 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:04.215084 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:04.216437 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:04.218121 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:04.218789 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:04.218918 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:04.223448 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:04.223462 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:04.223484 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:04.223530 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:04.227342 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:04.228147 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:04.228157 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:04.228166 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:04.228380 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:04.228751 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:04.228757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:04.228760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:04.229582 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:04.229823 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:04.230309 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 02:14:04.230805 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":323,"threads_started":1}
I20260926 02:14:04.232445 3004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec: No bootstrap required, opened a new log
I20260926 02:14:04.240513 3004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } }
I20260926 02:14:04.240928 3004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:04.241011 3004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248a9ebcadfd4363957532d4e5a993ec, State: Initialized, Role: FOLLOWER
I20260926 02:14:04.241593 3004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } }
I20260926 02:14:04.241799 3004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:14:04.241896 3004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:14:04.242026 3004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:04.244349 3004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } }
I20260926 02:14:04.244812 3004 leader_election.cc:302] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 248a9ebcadfd4363957532d4e5a993ec; no voters:
I20260926 02:14:04.245388 3004 leader_election.cc:288] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:14:04.245518 3009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:04.246155 3009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 1 LEADER]: Becoming Leader. State: Replica: 248a9ebcadfd4363957532d4e5a993ec, State: Running, Role: LEADER
I20260926 02:14:04.246681 3009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } }
I20260926 02:14:04.247404 3004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:14:04.249635 3010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } }
I20260926 02:14:04.249665 3011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 248a9ebcadfd4363957532d4e5a993ec. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } }
I20260926 02:14:04.250000 3011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:04.249980 3010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:04.254549 3015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:14:04.259800 3015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:14:04.269197 3015 catalog_manager.cc:1347] Generated new cluster ID: aa430a297210411e86ecefbf19f28df6
I20260926 02:14:04.269337 3015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:14:04.284116 3015 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:14:04.286154 3015 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:14:04.298425 3015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec: Generated new TSK 0
I20260926 02:14:04.299655 3015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab.
Entry for principal oryx/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab.
I20260926 02:14:04.368480 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34891
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab
--principal=oryx/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:04.699981 3030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:04.700289 3030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:04.700407 3030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:04.707777 3030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:04.707901 3030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:04.708048 3030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:14:04.717890 3030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab
--principal=oryx/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:04.719219 3030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:04.720579 3030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:04.733965 3036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:04.734015 3035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:04.734047 3038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:04.734961 3030 server_base.cc:1034] running on GCE node
I20260926 02:14:04.735602 3030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:04.736646 3030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:04.737856 3030 hybrid_clock.cc:648] HybridClock initialized: now 1790388844737805 us; error 46 us; skew 500 ppm
I20260926 02:14:04.738173 3030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:04 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388844, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:04.741288 3030 init.cc:377] Logged in from keytab as oryx/127.28.152.129@KRBTEST.COM (short username oryx)
I20260926 02:14:04.742772 3030 webserver.cc:460] Webserver started at http://127.28.152.129:41233/ using document root <none> and password file <none>
I20260926 02:14:04.743489 3030 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:04.743609 3030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:04.743896 3030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:04.747005 3030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "8e34eff83373477c97735ff40f839576"
format_stamp: "Formatted at 2026-09-26 02:14:04 on dist-test-slave-hllj"
server_key: "b51c46bda6f4412ec4407bbdc3f751af"
server_key_iv: "f30eea47cbe2a14a469102dffa41ae20"
server_key_version: "encryptionkey@0"
I20260926 02:14:04.747859 3030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "8e34eff83373477c97735ff40f839576"
format_stamp: "Formatted at 2026-09-26 02:14:04 on dist-test-slave-hllj"
server_key: "b51c46bda6f4412ec4407bbdc3f751af"
server_key_iv: "f30eea47cbe2a14a469102dffa41ae20"
server_key_version: "encryptionkey@0"
I20260926 02:14:04.753368 3030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 02:14:04.757555 3045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:04.759150 3030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:14:04.759366 3030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "8e34eff83373477c97735ff40f839576"
format_stamp: "Formatted at 2026-09-26 02:14:04 on dist-test-slave-hllj"
server_key: "b51c46bda6f4412ec4407bbdc3f751af"
server_key_iv: "f30eea47cbe2a14a469102dffa41ae20"
server_key_version: "encryptionkey@0"
I20260926 02:14:04.759563 3030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:04.783643 3030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:04.787990 3030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:04.788323 3030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:04.789374 3030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:04.790711 3030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:04.790786 3030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:04.790905 3030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:04.790958 3030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:04.820143 3030 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:40489
I20260926 02:14:04.820219 3157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:40489 every 8 connection(s)
I20260926 02:14:04.821502 3030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:14:04.828872 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 3030
I20260926 02:14:04.829157 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:14:04.829964 29282 external_mini_cluster.cc:1428] Setting key 9f366c978cde6b04ee6a5197e9dd7b85
Sep 26 02:14:04 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388844, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:04.842713 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:04.823977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:58871 (local address 127.28.152.190:34891)
0926 02:14:04.824414 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:04.824419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:04.825467 (+ 1048us) server_negotiation.cc:407] Connection header received
0926 02:14:04.827197 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:04.827201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:04.827266 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:04.827358 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:04.828893 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:04.829970 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:04.831986 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:04.832371 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:04.832464 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:04.836896 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:04.836902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:04.836905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:04.836942 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:04.839437 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:04.840302 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:04.840306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:04.840308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:04.840427 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:04.840926 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:04.840930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:04.840932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:04.841289 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:04.841408 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:04.842277 (+ 869us) server_negotiation.cc:299] Negotiation successful
0926 02:14:04.842455 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":218,"threads_started":1}
I20260926 02:14:04.844167 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:04.824574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.129:58871)
0926 02:14:04.825317 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:04.825358 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:04.826951 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:04.827560 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:04.827569 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:04.828087 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:04.828710 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:04.828727 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:04.830179 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:04.830184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:04.831838 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:04.831854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:04.832609 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:04.832616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:04.832900 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:04.833619 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:04.833649 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:04.836532 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:04.839570 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:04.839585 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:04.839605 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:04.840189 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:04.840524 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:04.840528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:04.840532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:04.840795 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:04.841412 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:04.841421 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:04.841883 (+ 462us) client_negotiation.cc:769] Sending connection context
0926 02:14:04.842184 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 02:14:04.842755 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":238,"threads_started":1}
I20260926 02:14:04.845849 3158 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34891
I20260926 02:14:04.846172 3158 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:04.846925 3158 heartbeater.cc:511] Master 127.28.152.190:34891 requested a full tablet report, sending...
I20260926 02:14:04.849309 2969 ts_manager.cc:194] Registered new tserver with Master: 8e34eff83373477c97735ff40f839576 (127.28.152.129:40489)
I20260926 02:14:04.850756 2969 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:58871
WARNING: no policy specified for oryx/127.28.152.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab.
Entry for principal oryx/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab.
I20260926 02:14:04.876909 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34891
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab
--principal=oryx/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:05.202584 3164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:05.202888 3164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:05.203016 3164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:05.214383 3164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:05.214529 3164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:05.214710 3164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:14:05.229009 3164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab
--principal=oryx/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:05.230384 3164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:05.231732 3164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:05.245043 3169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:05.245066 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:05.245108 3172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:05.245817 3164 server_base.cc:1034] running on GCE node
I20260926 02:14:05.246461 3164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:05.247434 3164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:05.248626 3164 hybrid_clock.cc:648] HybridClock initialized: now 1790388845248574 us; error 47 us; skew 500 ppm
I20260926 02:14:05.248930 3164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:05 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388845, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:05.252279 3164 init.cc:377] Logged in from keytab as oryx/127.28.152.130@KRBTEST.COM (short username oryx)
I20260926 02:14:05.253885 3164 webserver.cc:460] Webserver started at http://127.28.152.130:41601/ using document root <none> and password file <none>
I20260926 02:14:05.254638 3164 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:05.254765 3164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:05.255060 3164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:05.258173 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "c49afa0c59cd4109bd029349db882ecd"
format_stamp: "Formatted at 2026-09-26 02:14:05 on dist-test-slave-hllj"
server_key: "119793e138150ad9049d31c8c6bb3d5d"
server_key_iv: "0838d5b54613b0cb1c640f872351d244"
server_key_version: "encryptionkey@0"
I20260926 02:14:05.258975 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "c49afa0c59cd4109bd029349db882ecd"
format_stamp: "Formatted at 2026-09-26 02:14:05 on dist-test-slave-hllj"
server_key: "119793e138150ad9049d31c8c6bb3d5d"
server_key_iv: "0838d5b54613b0cb1c640f872351d244"
server_key_version: "encryptionkey@0"
I20260926 02:14:05.264261 3164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:14:05.267987 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:05.269558 3164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:14:05.269719 3164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "c49afa0c59cd4109bd029349db882ecd"
format_stamp: "Formatted at 2026-09-26 02:14:05 on dist-test-slave-hllj"
server_key: "119793e138150ad9049d31c8c6bb3d5d"
server_key_iv: "0838d5b54613b0cb1c640f872351d244"
server_key_version: "encryptionkey@0"
I20260926 02:14:05.269913 3164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:05.294099 3164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:05.298291 3164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:05.298580 3164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:05.299572 3164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:05.300932 3164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:05.301005 3164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:05.301155 3164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:05.301219 3164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 02:14:05.332418 3164 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:34051
I20260926 02:14:05.332456 3291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:34051 every 8 connection(s)
I20260926 02:14:05.333777 3164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:14:05.337217 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 3164
I20260926 02:14:05.337440 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:14:05.337987 29282 external_mini_cluster.cc:1428] Setting key 3bbdb9cb123f20f32eb71be2ec911777
Sep 26 02:14:05 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388845, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:05.354915 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:05.336378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:36159 (local address 127.28.152.190:34891)
0926 02:14:05.336551 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:05.336557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:05.338153 (+ 1596us) server_negotiation.cc:407] Connection header received
0926 02:14:05.340330 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:05.340335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:05.340415 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:05.340517 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:05.342126 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:05.343050 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:05.344655 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:05.345215 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:05.345341 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:05.349154 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:05.349162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:05.349167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:05.349204 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:05.351695 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:05.352521 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:05.352525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:05.352527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:05.352650 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:05.353105 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:05.353108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:05.353110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:05.353416 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:05.353523 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:05.354447 (+ 924us) server_negotiation.cc:299] Negotiation successful
0926 02:14:05.354623 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:14:05.356419 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:05.336987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.130:36159)
0926 02:14:05.337971 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:05.338020 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:05.340060 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:05.340745 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:05.340754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:05.341359 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:05.341965 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:05.341983 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:05.343191 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:05.343196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:05.344519 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:05.344532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:05.345456 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:05.345464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:05.345696 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:05.346368 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:05.346391 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:05.348898 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:05.351837 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:05.351846 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:05.351864 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:05.352419 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:05.352747 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:05.352752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:05.352756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:05.352999 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:05.353534 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:05.353542 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:05.354034 (+ 492us) client_negotiation.cc:769] Sending connection context
0926 02:14:05.354359 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 02:14:05.354964 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":362,"threads_started":1}
I20260926 02:14:05.357913 3292 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34891
WARNING: no policy specified for oryx/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:14:05.358286 3292 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:05.359100 3292 heartbeater.cc:511] Master 127.28.152.190:34891 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.131@KRBTEST.COM" created.
I20260926 02:14:05.360605 2969 ts_manager.cc:194] Registered new tserver with Master: c49afa0c59cd4109bd029349db882ecd (127.28.152.130:34051)
I20260926 02:14:05.361752 2969 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:36159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab.
Entry for principal oryx/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab.
I20260926 02:14:05.384310 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34891
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab
--principal=oryx/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:05.712760 3297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:05.713078 3297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:05.713254 3297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:05.720606 3297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:05.720755 3297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:05.720902 3297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:14:05.731120 3297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab
--principal=oryx/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:05.732501 3297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:05.733893 3297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:05.746851 3302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:05.746937 3303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:05.747089 3305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:05.748034 3297 server_base.cc:1034] running on GCE node
I20260926 02:14:05.748663 3297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:05.749660 3297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:05.750880 3297 hybrid_clock.cc:648] HybridClock initialized: now 1790388845750829 us; error 37 us; skew 500 ppm
I20260926 02:14:05.751186 3297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:05 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388845, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:05.754408 3297 init.cc:377] Logged in from keytab as oryx/127.28.152.131@KRBTEST.COM (short username oryx)
I20260926 02:14:05.755937 3297 webserver.cc:460] Webserver started at http://127.28.152.131:38947/ using document root <none> and password file <none>
I20260926 02:14:05.756723 3297 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:05.756836 3297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:05.757154 3297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:05.760185 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "320e0037c6754a51b79146fc5f118f45"
format_stamp: "Formatted at 2026-09-26 02:14:05 on dist-test-slave-hllj"
server_key: "f33e6c6efe82231e46ed41bd18b3e46a"
server_key_iv: "c22dd69c836432051591d6753b6c25e9"
server_key_version: "encryptionkey@0"
I20260926 02:14:05.760978 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "320e0037c6754a51b79146fc5f118f45"
format_stamp: "Formatted at 2026-09-26 02:14:05 on dist-test-slave-hllj"
server_key: "f33e6c6efe82231e46ed41bd18b3e46a"
server_key_iv: "c22dd69c836432051591d6753b6c25e9"
server_key_version: "encryptionkey@0"
I20260926 02:14:05.766567 3297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 02:14:05.770385 3312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:05.772399 3297 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 02:14:05.772583 3297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "320e0037c6754a51b79146fc5f118f45"
format_stamp: "Formatted at 2026-09-26 02:14:05 on dist-test-slave-hllj"
server_key: "f33e6c6efe82231e46ed41bd18b3e46a"
server_key_iv: "c22dd69c836432051591d6753b6c25e9"
server_key_version: "encryptionkey@0"
I20260926 02:14:05.772827 3297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:05.786032 3297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:05.790370 3297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:05.790647 3297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:05.791630 3297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:05.793023 3297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:05.793105 3297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:05.793247 3297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:05.793304 3297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:05.822968 3297 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:36667
I20260926 02:14:05.822999 3424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:36667 every 8 connection(s)
I20260926 02:14:05.824350 3297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:14:05.834585 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 3297
I20260926 02:14:05.834808 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:14:05.835251 29282 external_mini_cluster.cc:1428] Setting key d9144644d4a809346cc76b973299ce40
Sep 26 02:14:05 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388845, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:05.845300 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:05.827114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:38925 (local address 127.28.152.190:34891)
0926 02:14:05.827312 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:05.827317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:05.828552 (+ 1235us) server_negotiation.cc:407] Connection header received
0926 02:14:05.830526 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:05.830532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:05.830643 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:05.830732 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:05.832403 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:05.833216 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:05.834820 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:05.835262 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:05.835397 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:05.839553 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:05.839559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:05.839562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:05.839597 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:05.842212 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:05.842973 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:05.842977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:05.842980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:05.843116 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:05.843521 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:05.843525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:05.843527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:05.843809 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:05.843939 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:05.844673 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 02:14:05.844908 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:14:05.846319 3427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:05.827608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.131:38925)
0926 02:14:05.828365 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:05.828404 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:05.830211 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:05.831024 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:05.831033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:05.831577 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:05.832211 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:05.832230 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:05.833374 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:05.833378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:05.834660 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:05.834673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:05.835496 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:05.835504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:05.835792 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:05.836555 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:05.836597 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:05.839327 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:05.842382 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:05.842393 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:05.842413 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:05.842852 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:05.843217 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:05.843222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:05.843226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:05.843417 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:05.843932 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:05.843941 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:05.844309 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 02:14:05.844560 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 02:14:05.845169 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":299,"threads_started":1}
I20260926 02:14:05.847823 3425 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34891
I20260926 02:14:05.848126 3425 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:05.848927 3425 heartbeater.cc:511] Master 127.28.152.190:34891 requested a full tablet report, sending...
I20260926 02:14:05.850740 2969 ts_manager.cc:194] Registered new tserver with Master: 320e0037c6754a51b79146fc5f118f45 (127.28.152.131:36667)
I20260926 02:14:05.851588 2969 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:38925
I20260926 02:14:05.853750 3158 heartbeater.cc:503] Master 127.28.152.190:34891 was elected leader, sending a full tablet report...
I20260926 02:14:05.863838 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.152.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.189.keytab.
Entry for principal oryx/127.28.152.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.189.keytab.
I20260926 02:14:05.912921 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.189:37903
--webserver_interface=127.28.152.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.189.keytab
--principal=oryx/127.28.152.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34891,127.28.152.189:37903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:06.235230 3430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:06.235534 3430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:06.235637 3430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:06.243135 3430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:14:06.243250 3430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:06.243307 3430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:06.243384 3430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:14:06.243441 3430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:14:06.253476 3430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34891,127.28.152.189:37903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.189.keytab
--principal=oryx/127.28.152.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.189:37903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.152.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
W20260926 02:14:06.253849 3430 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.152.190:34891,127.28.152.189:37903'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 02:14:06.254954 3430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:06.256690 3430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:06.269178 3436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:06.269199 3438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:06.269272 3435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:06.269997 3430 server_base.cc:1034] running on GCE node
I20260926 02:14:06.270884 3430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:06.272364 3430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:06.273572 3430 hybrid_clock.cc:648] HybridClock initialized: now 1790388846273509 us; error 48 us; skew 500 ppm
I20260926 02:14:06.273913 3430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:06 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388846, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:06.277189 3430 init.cc:377] Logged in from keytab as oryx/127.28.152.189@KRBTEST.COM (short username oryx)
I20260926 02:14:06.278822 3430 webserver.cc:460] Webserver started at http://127.28.152.189:42687/ using document root <none> and password file <none>
I20260926 02:14:06.279580 3430 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:06.279736 3430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:06.280048 3430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:06.283213 3430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data/instance:
uuid: "f31a651ed7d94d5caa654cd28802f45e"
format_stamp: "Formatted at 2026-09-26 02:14:06 on dist-test-slave-hllj"
server_key: "44d2daadc40c5de8c901a3d38fc4b909"
server_key_iv: "369787b5b089bde5470000fe6dc40722"
server_key_version: "encryptionkey@0"
I20260926 02:14:06.284085 3430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/wal/instance:
uuid: "f31a651ed7d94d5caa654cd28802f45e"
format_stamp: "Formatted at 2026-09-26 02:14:06 on dist-test-slave-hllj"
server_key: "44d2daadc40c5de8c901a3d38fc4b909"
server_key_iv: "369787b5b089bde5470000fe6dc40722"
server_key_version: "encryptionkey@0"
I20260926 02:14:06.289647 3430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:14:06.293455 3445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:06.294984 3430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:14:06.295171 3430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/wal
uuid: "f31a651ed7d94d5caa654cd28802f45e"
format_stamp: "Formatted at 2026-09-26 02:14:06 on dist-test-slave-hllj"
server_key: "44d2daadc40c5de8c901a3d38fc4b909"
server_key_iv: "369787b5b089bde5470000fe6dc40722"
server_key_version: "encryptionkey@0"
I20260926 02:14:06.295392 3430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:06.308894 3430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:06.313416 3430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:06.313678 3430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:06.335716 3430 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.189:37903
I20260926 02:14:06.335753 3496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.189:37903 every 8 connection(s)
I20260926 02:14:06.337121 3430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
I20260926 02:14:06.341027 3497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:06.342321 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 3430
I20260926 02:14:06.350366 3497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:14:06 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.152.189@KRBTEST.COM
Sep 26 02:14:06 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388846, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.189@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:06.364876 3292 heartbeater.cc:503] Master 127.28.152.190:34891 was elected leader, sending a full tablet report...
I20260926 02:14:06.366016 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.344455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.28.152.189:37903)
0926 02:14:06.345262 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:06.345278 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:06.345331 (+ 53us) server_negotiation.cc:407] Connection header received
0926 02:14:06.347232 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:06.347263 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:06.347864 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:06.348348 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:06.349809 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.351759 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.353097 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.354008 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.354135 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.358495 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:06.358504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:06.358516 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:06.358556 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:06.361535 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.362197 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.362204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.362212 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.362356 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.362720 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.362725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.362727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.363308 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:06.363566 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:06.364071 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 02:14:06.364513 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":319,"threads_started":1}
I20260926 02:14:06.367722 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.353373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.28.152.190:34891)
0926 02:14:06.353820 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:06.353825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:06.354830 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 02:14:06.354986 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:06.354991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:06.355069 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:06.355162 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:06.356249 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.357013 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.358394 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.358795 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.358905 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.362024 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:06.362029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:06.362032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:06.362064 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:06.364709 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.365516 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.365521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.365525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.365684 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.366128 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.366132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.366135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.366518 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:06.366660 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:06.367169 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 02:14:06.367372 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20260926 02:14:06.367774 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.353361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:43022)
0926 02:14:06.354685 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:06.354736 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:06.354882 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:06.355234 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:06.355239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:06.355889 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:06.356109 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.356116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.357183 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.357188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.358257 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.358269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.358985 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.358991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.359153 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.359691 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:06.359711 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:06.361847 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:06.364821 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.364849 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.364857 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.365409 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.365815 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.365820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.365823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.365978 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.366728 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:06.366736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:06.366878 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 02:14:06.367035 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 02:14:06.367366 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"thread_start_us":1005,"threads_started":1}
I20260926 02:14:06.369865 3497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:14:06 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388846, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.189@KRBTEST.COM for oryx/127.28.152.189@KRBTEST.COM
I20260926 02:14:06.381419 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.370591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:37903 (local address 127.0.0.1:46342)
0926 02:14:06.370766 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:06.370784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:06.370885 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:06.371118 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:06.371120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:06.371451 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:06.371582 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.371589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.372483 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.372486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.373428 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.373439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.374036 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.374040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.374194 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.374730 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:06.374746 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:06.376627 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:06.379278 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.379283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.379288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.379616 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.379915 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.379921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.379924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.380041 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.380507 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:06.380513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:06.380639 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 02:14:06.380753 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:14:06.381034 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 02:14:06.381498 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.370803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.28.152.189:37903)
0926 02:14:06.370924 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:06.370928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:06.370941 (+ 13us) server_negotiation.cc:407] Connection header received
0926 02:14:06.370975 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:06.370978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:06.371026 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:06.371097 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:06.371683 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.372372 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.373527 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.373917 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.374001 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.376833 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:06.376838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:06.376841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:06.376866 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:06.379152 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.379717 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.379722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.379725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.379826 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.380128 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.380133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.380136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.380409 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:06.380526 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:06.380831 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 02:14:06.381201 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":39}
I20260926 02:14:06.386615 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: Bootstrap starting.
I20260926 02:14:06.390241 3497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:06.391255 3497 log.cc:824] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:06.394068 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: No bootstrap required, opened a new log
I20260926 02:14:06.401300 3497 raft_consensus.cc:348] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } }
I20260926 02:14:06.401660 3497 raft_consensus.cc:374] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:06.401738 3497 raft_consensus.cc:729] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f31a651ed7d94d5caa654cd28802f45e, State: Initialized, Role: FOLLOWER
I20260926 02:14:06.402341 3497 consensus_queue.cc:260] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } }
I20260926 02:14:06.403165 3504 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } } }
I20260926 02:14:06.403451 3504 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:14:06.404224 3497 sys_catalog.cc:564] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:14:06.413162 3515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:14:06.413339 3515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:14:06.416290 3430 master_runner.cc:428] Detected that this master f31a651ed7d94d5caa654cd28802f45e is joining an existing cluster
I20260926 02:14:06.416416 3430 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.152.189:37903
I20260926 02:14:06.440765 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.425373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.28.152.189:37903)
0926 02:14:06.425564 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:06.425570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:06.426783 (+ 1213us) server_negotiation.cc:407] Connection header received
0926 02:14:06.426994 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:06.426998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:06.427063 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:06.427138 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:06.428436 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.429560 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.431429 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.432124 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.432221 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.434470 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:06.434476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:06.434480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:06.434509 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:06.437558 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.438511 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.438516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.438520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.438670 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.439080 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.439084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.439088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.439443 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:06.439580 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:06.440148 (+ 568us) server_negotiation.cc:299] Negotiation successful
0926 02:14:06.440395 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":66}
I20260926 02:14:06.440816 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.420923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.28.152.190:34891)
0926 02:14:06.421123 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:06.421164 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:06.426195 (+ 5031us) server_negotiation.cc:407] Connection header received
0926 02:14:06.426284 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:06.426292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:06.426392 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:06.426508 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:06.427909 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.428948 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.430831 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.431396 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.431485 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.434078 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:06.434085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:06.434090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:06.434143 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:06.437516 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.438459 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.438466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.438470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.438629 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.439076 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.439081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.439085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.439489 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:06.439622 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:06.440231 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 02:14:06.440479 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for oryx/127.28.152.188@KRBTEST.COM; defaulting to no policy
I20260926 02:14:06.441874 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.420435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:43032)
0926 02:14:06.425989 (+ 5554us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:06.426007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:06.426170 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:06.426526 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:06.426530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:06.427188 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:06.427717 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.427729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.429113 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.429118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.430671 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.430683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.431761 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.431767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.431952 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.432631 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:06.432658 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:06.433847 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:06.437693 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.437699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.437703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.438312 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.438775 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.438780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.438783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.438949 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.439699 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:06.439703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:06.439918 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 02:14:06.441081 (+ 1163us) client_negotiation.cc:240] Negotiation successful
0926 02:14:06.441444 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5428,"mutex_wait_us":22,"thread_start_us":5287,"threads_started":1}
I20260926 02:14:06.442396 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.420573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:37903 (local address 127.0.0.1:46350)
0926 02:14:06.426694 (+ 6121us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:06.426710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:06.426865 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:06.427326 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:06.427330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:06.427967 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:06.428283 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.428295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.429763 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.429768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.431292 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.431306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.432513 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.432519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.432691 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.433272 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:06.433296 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:06.434300 (+ 1004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:06.437696 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.437701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.437705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.438390 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.438802 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.438806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.438810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.438974 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.439662 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:06.439665 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:06.439807 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 02:14:06.441532 (+ 1725us) client_negotiation.cc:240] Negotiation successful
0926 02:14:06.441821 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5984,"spinlock_wait_cycles":42240,"thread_start_us":5781,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.188@KRBTEST.COM" created.
Sep 26 02:14:06 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388844, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.190@KRBTEST.COM for oryx/127.28.152.189@KRBTEST.COM
I20260926 02:14:06.469800 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.448981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.28.152.189:37903)
0926 02:14:06.449209 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:06.449216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:06.452506 (+ 3290us) server_negotiation.cc:407] Connection header received
0926 02:14:06.452892 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:06.452897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:06.452987 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:06.453090 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:06.454689 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.455756 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.457779 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.458458 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.458579 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.463039 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:06.463048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:06.463052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:06.463102 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:06.466491 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.467511 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.467516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.467518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.467653 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.468050 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.468053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.468056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.468389 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:06.468567 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:06.469295 (+ 728us) server_negotiation.cc:299] Negotiation successful
0926 02:14:06.469486 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:14:06.469837 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.448951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:37903 (local address 127.0.0.1:46360)
0926 02:14:06.452349 (+ 3398us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:06.452391 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:06.452660 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:06.453215 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:06.453225 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:06.454143 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:06.454455 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.454469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.455892 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.455898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.457594 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.457612 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.458643 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.458650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.458880 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.459712 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:06.459738 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:06.462759 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:06.466687 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.466701 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.466710 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.467360 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.467761 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.467767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.467771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.467947 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.468537 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:06.468547 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:06.468760 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 02:14:06.468985 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 02:14:06.469405 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3140,"thread_start_us":2968,"threads_started":1}
I20260926 02:14:06.471892 2969 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.152.189:37903: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e"
member_type: NON_VOTER
last_known_addr {
host: "127.28.152.189"
port: 37903
}
attrs {
promote: true
}
}
dest_uuid: "248a9ebcadfd4363957532d4e5a993ec"
cas_config_opid_index: -1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.188.keytab.
Entry for principal oryx/127.28.152.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.188.keytab.
I20260926 02:14:06.473186 2969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } }
I20260926 02:14:06.474674 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.188:45997
--webserver_interface=127.28.152.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.188.keytab
--principal=oryx/127.28.152.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34891,127.28.152.189:37903,127.28.152.188:45997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:14:06.484560 3528 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f31a651ed7d94d5caa654cd28802f45e (127.28.152.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } } }
I20260926 02:14:06.487064 3472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:06.489774 3528 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.152.189:37903
I20260926 02:14:06.490552 3430 master.cc:537] Master@127.28.152.189:37903 shutting down...
I20260926 02:14:06.490358 3527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } } }
I20260926 02:14:06.490794 3527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:06.496227 3472 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 FOLLOWER]: Refusing update from remote peer 248a9ebcadfd4363957532d4e5a993ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 02:14:06.497628 3527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:14:06.500916 2956 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 02:14:06.504169 3430 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:06.504663 3430 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:06.504877 3430 tablet_replica.cc:330] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: stopping tablet replica
W20260926 02:14:06.508792 2956 consensus_peers.cc:487] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec -> Peer f31a651ed7d94d5caa654cd28802f45e (127.28.152.189:37903): Couldn't send request to peer f31a651ed7d94d5caa654cd28802f45e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 02:14:06.522058 3430 master.cc:552] Master@127.28.152.189:37903 shutdown complete.
I20260926 02:14:06.522150 3430 master_runner.cc:305] Clearing existing system tablet
I20260926 02:14:06.522511 3430 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:06.522612 3430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:06.524749 3430 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 02:14:06.526167 3533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:06.526463 3430 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:06.526733 3430 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 02:14:06.526814 3430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/wal
uuid: "f31a651ed7d94d5caa654cd28802f45e"
format_stamp: "Formatted at 2026-09-26 02:14:06 on dist-test-slave-hllj"
server_key: "44d2daadc40c5de8c901a3d38fc4b909"
server_key_iv: "369787b5b089bde5470000fe6dc40722"
server_key_version: "encryptionkey@0"
I20260926 02:14:06.530216 3430 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:06.535038 3430 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 02:14:06.535224 3430 log.cc:1196] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 02:14:06.535601 3430 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: Deleting consensus metadata
I20260926 02:14:06.535820 3430 master_runner.cc:315] Copying system tablet from 127.28.152.190:34891
I20260926 02:14:06.538115 3430 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: tablet copy: Beginning tablet copy session from remote peer at address 127.28.152.190:34891
I20260926 02:14:06.550552 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.538830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.28.152.190:34891)
0926 02:14:06.539030 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:06.539037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:06.539460 (+ 423us) server_negotiation.cc:407] Connection header received
0926 02:14:06.539655 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:06.539661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:06.539730 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:06.539812 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:06.540758 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.541681 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.543059 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.543478 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.543589 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.545353 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:06.545358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:06.545361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:06.545394 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:06.547960 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.548647 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.548654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.548657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.548786 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:06.549151 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:06.549157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:06.549160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:06.549437 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:06.549567 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:06.549941 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 02:14:06.550220 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:14:06.550570 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.538751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:43040)
0926 02:14:06.539344 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:06.539371 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:06.539511 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:06.539859 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:06.539864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:06.540362 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:06.540603 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.540613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.541815 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.541821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.542910 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.542923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.543630 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.543635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.543807 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.544397 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:06.544417 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:06.545122 (+ 705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:06.548124 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.548129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.548137 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.548546 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.548887 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:06.548892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:06.548895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:06.549030 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:06.549549 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:06.549554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:06.549716 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 02:14:06.549834 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 02:14:06.550135 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":393,"threads_started":1}
I20260926 02:14:06.551765 2989 tablet_copy_service.cc:140] P 248a9ebcadfd4363957532d4e5a993ec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f31a651ed7d94d5caa654cd28802f45e ({username='oryx', principal='oryx/127.28.152.189@KRBTEST.COM'} at 127.0.0.1:43040)
I20260926 02:14:06.552071 2989 tablet_copy_service.cc:161] P 248a9ebcadfd4363957532d4e5a993ec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f31a651ed7d94d5caa654cd28802f45e at {username='oryx', principal='oryx/127.28.152.189@KRBTEST.COM'} at 127.0.0.1:43040: session id = f31a651ed7d94d5caa654cd28802f45e-00000000000000000000000000000000
I20260926 02:14:06.556433 2989 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 02:14:06.558982 3430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 02:14:06.565874 3430 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: tablet copy: Starting download of 0 data blocks...
I20260926 02:14:06.566236 3430 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: tablet copy: Starting download of 1 WAL segments...
I20260926 02:14:06.569322 3430 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 02:14:06.574031 2989 tablet_copy_service.cc:342] P 248a9ebcadfd4363957532d4e5a993ec: Request end of tablet copy session f31a651ed7d94d5caa654cd28802f45e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.152.189@KRBTEST.COM'} at 127.0.0.1:43040
I20260926 02:14:06.574258 2989 tablet_copy_service.cc:433] P 248a9ebcadfd4363957532d4e5a993ec: ending tablet copy session f31a651ed7d94d5caa654cd28802f45e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f31a651ed7d94d5caa654cd28802f45e
I20260926 02:14:06.587781 3430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:06.590675 3540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:06.594781 3541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:06.602409 3430 server_base.cc:1034] running on GCE node
W20260926 02:14:06.602679 3543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:06.603163 3430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:06.603860 3430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:06.605089 3430 hybrid_clock.cc:648] HybridClock initialized: now 1790388846605016 us; error 47 us; skew 500 ppm
I20260926 02:14:06.605350 3430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:06 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388846, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:06.607564 3430 init.cc:377] Logged in from keytab as oryx/127.28.152.189@KRBTEST.COM (short username oryx)
I20260926 02:14:06.614180 3430 webserver.cc:460] Webserver started at http://127.28.152.189:44383/ using document root <none> and password file <none>
I20260926 02:14:06.614636 3430 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:06.614738 3430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:06.618073 3430 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260926 02:14:06.619930 3550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:06.620355 3430 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 02:14:06.620497 3430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/wal
uuid: "f31a651ed7d94d5caa654cd28802f45e"
format_stamp: "Formatted at 2026-09-26 02:14:06 on dist-test-slave-hllj"
server_key: "44d2daadc40c5de8c901a3d38fc4b909"
server_key_iv: "369787b5b089bde5470000fe6dc40722"
server_key_version: "encryptionkey@0"
I20260926 02:14:06.620674 3430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:06.653399 3430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:06.657500 3430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:06.682314 3430 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.189:37903
I20260926 02:14:06.682343 3601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.189:37903 every 8 connection(s)
I20260926 02:14:06.683507 3430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
I20260926 02:14:06.689080 3602 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:14:06.693120 3602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: Bootstrap starting.
I20260926 02:14:06.710925 3602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:14:06.711452 3602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: Bootstrap complete.
I20260926 02:14:06.712528 3602 raft_consensus.cc:348] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } }
I20260926 02:14:06.712809 3602 raft_consensus.cc:729] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f31a651ed7d94d5caa654cd28802f45e, State: Initialized, Role: LEARNER
I20260926 02:14:06.713048 3602 consensus_queue.cc:260] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } }
I20260926 02:14:06.713922 3605 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } } }
I20260926 02:14:06.714215 3602 sys_catalog.cc:564] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:14:06.714341 3605 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: This master's current role is: LEARNER
I20260926 02:14:06.725914 3616 catalog_manager.cc:1259] Loaded cluster ID: aa430a297210411e86ecefbf19f28df6
I20260926 02:14:06.726035 3616 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: loading cluster ID for follower catalog manager: success
I20260926 02:14:06.730000 3616 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: acquiring CA information for follower catalog manager: success
I20260926 02:14:06.732990 3616 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:14:06.854417 3425 heartbeater.cc:503] Master 127.28.152.190:34891 was elected leader, sending a full tablet report...
W20260926 02:14:06.883942 3529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:06.884289 3529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:06.884433 3529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:06.891949 3529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:14:06.892077 3529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:06.892122 3529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:06.892179 3529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:14:06.892222 3529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:14:06.902328 3529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34891,127.28.152.189:37903,127.28.152.188:45997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.188.keytab
--principal=oryx/127.28.152.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.188:45997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.152.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:06.903622 3529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:06.904947 3529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:06.917454 3622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:06.917375 3624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:06.917385 3621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:06.918191 3529 server_base.cc:1034] running on GCE node
I20260926 02:14:06.918995 3529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:06.920490 3529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:06.921689 3529 hybrid_clock.cc:648] HybridClock initialized: now 1790388846921648 us; error 37 us; skew 500 ppm
I20260926 02:14:06.922060 3529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:06 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388846, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:06.925587 3529 init.cc:377] Logged in from keytab as oryx/127.28.152.188@KRBTEST.COM (short username oryx)
I20260926 02:14:06.927175 3529 webserver.cc:460] Webserver started at http://127.28.152.188:36717/ using document root <none> and password file <none>
I20260926 02:14:06.927881 3529 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:06.928001 3529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:06.928296 3529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:06.931406 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data/instance:
uuid: "40341a9abb464c30ae76521f28ee3be3"
format_stamp: "Formatted at 2026-09-26 02:14:06 on dist-test-slave-hllj"
server_key: "78cbd2f13841b9bdef8d7442e7ea644a"
server_key_iv: "cc2117f2219d2c1f4fef64b3b5f26145"
server_key_version: "encryptionkey@0"
I20260926 02:14:06.932268 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/wal/instance:
uuid: "40341a9abb464c30ae76521f28ee3be3"
format_stamp: "Formatted at 2026-09-26 02:14:06 on dist-test-slave-hllj"
server_key: "78cbd2f13841b9bdef8d7442e7ea644a"
server_key_iv: "cc2117f2219d2c1f4fef64b3b5f26145"
server_key_version: "encryptionkey@0"
I20260926 02:14:06.937914 3529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:14:06.941502 3631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:06.943051 3529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 02:14:06.943199 3529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/wal
uuid: "40341a9abb464c30ae76521f28ee3be3"
format_stamp: "Formatted at 2026-09-26 02:14:06 on dist-test-slave-hllj"
server_key: "78cbd2f13841b9bdef8d7442e7ea644a"
server_key_iv: "cc2117f2219d2c1f4fef64b3b5f26145"
server_key_version: "encryptionkey@0"
I20260926 02:14:06.943437 3529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:06.962061 3529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:06.967458 3529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:06.967727 3529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:06.986960 3529 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.188:45997
I20260926 02:14:06.986976 3682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.188:45997 every 8 connection(s)
I20260926 02:14:06.988942 3529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
I20260926 02:14:06.993439 3685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:06.994047 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.986805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:37903 (local address 127.0.0.1:46370)
0926 02:14:06.987999 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:06.988019 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:06.988184 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:06.988549 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:06.988553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:06.988590 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:06.988796 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.988808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.990562 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.990567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.991856 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:06.991869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.993378 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.993383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:06.993531 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.993601 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 02:14:06.993689 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 02:14:06.993814 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"thread_start_us":440,"threads_started":1}
I20260926 02:14:06.994175 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.986945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.28.152.189:37903)
0926 02:14:06.987840 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:06.987844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:06.988115 (+ 271us) server_negotiation.cc:407] Connection header received
0926 02:14:06.988305 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:06.988309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:06.988379 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:06.988488 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:06.989010 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.990398 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.992028 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:06.993200 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:06.993384 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:06.993579 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:06.993730 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 02:14:06.993878 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":626,"threads_started":1}
I20260926 02:14:06.995152 3577 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 02:14:06.995709 3605 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 248a9ebcadfd4363957532d4e5a993ec. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } } }
I20260926 02:14:06.995959 3605 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: This master's current role is: LEARNER
I20260926 02:14:06.997020 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 3529
I20260926 02:14:07.002449 3685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388846, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.188@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.152.188@KRBTEST.COM
I20260926 02:14:07.021795 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.005661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.28.152.190:34891)
0926 02:14:07.005855 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.005860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.006322 (+ 462us) server_negotiation.cc:407] Connection header received
0926 02:14:07.006483 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.006487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.006549 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.006625 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.007984 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.009037 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.010949 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.011464 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.011591 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.015262 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.015269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.015274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.015318 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.018307 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.019206 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.019211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.019214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.019348 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.019799 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.019804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.019806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.020195 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.020393 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.021320 (+ 927us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.021527 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 02:14:07.023712 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:06.999523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.28.152.188:45997)
0926 02:14:07.000390 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.000404 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.000448 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:14:07.002560 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.002590 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.003241 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.003729 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.005539 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.007551 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.009420 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.010448 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.010615 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.015884 (+ 5269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.015893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.015908 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.015938 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.019619 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.020528 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.020536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.020539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.020690 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.021120 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.021143 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.021146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.021662 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.021975 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.022327 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.022559 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":362,"threads_started":1}
I20260926 02:14:07.023725 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.005449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:43044)
0926 02:14:07.006170 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.006220 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.006376 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.006710 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.006717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.007499 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:07.007854 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.007863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.009229 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.009233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.010799 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.010811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.011659 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.011663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.011807 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.012415 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:07.012443 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:07.015028 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:07.018447 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.018457 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.018473 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.019095 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.019452 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.019457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.019460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.019689 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.020371 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:07.020378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:07.020872 (+ 494us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.021234 (+ 362us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.022207 (+ 973us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":1,"thread_start_us":294,"threads_started":1}
I20260926 02:14:07.025724 3685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388846, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.188@KRBTEST.COM for oryx/127.28.152.189@KRBTEST.COM
I20260926 02:14:07.039436 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.026596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.28.152.189:37903)
0926 02:14:07.026763 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.026769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.026956 (+ 187us) server_negotiation.cc:407] Connection header received
0926 02:14:07.027096 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.027100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.027163 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.027245 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.027883 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.028575 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.029847 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.030254 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.030341 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.033359 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.033364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.033368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.033403 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.036347 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.037195 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.037201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.037205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.037366 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.037797 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.037800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.037802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.038091 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.038303 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.038814 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.039093 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 02:14:07.039722 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.026530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:37903 (local address 127.0.0.1:46386)
0926 02:14:07.026861 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.026879 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.026997 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.027311 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.027315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.027631 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:07.027774 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.027782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.028713 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.028717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.029721 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.029732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.030393 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.030397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.030540 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.031082 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:07.031097 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:07.033121 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:07.036542 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.036547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.036551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.037008 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.037511 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.037515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.037518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.037679 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.038246 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:07.038254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:07.038420 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.038634 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.039004 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 02:14:07.040817 3685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388846, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.188@KRBTEST.COM for oryx/127.28.152.188@KRBTEST.COM
I20260926 02:14:07.053241 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.041656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:45997 (local address 127.0.0.1:60282)
0926 02:14:07.041875 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.041890 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.041987 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.042272 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.042275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.042634 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:07.042824 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.042836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.043868 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.043872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.044789 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.044802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.045564 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.045568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.045740 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.046315 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:07.046331 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:07.048438 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:07.051133 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.051137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.051140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.051526 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.051821 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.051825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.051827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.051976 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.052456 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:07.052459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:07.052567 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.052683 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.052888 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 02:14:07.053308 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.041648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.28.152.188:45997)
0926 02:14:07.041793 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.041798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.041931 (+ 133us) server_negotiation.cc:407] Connection header received
0926 02:14:07.042101 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.042105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.042158 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.042216 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.042982 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.043727 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.044942 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.045431 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.045536 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.048614 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.048619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.048622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.048657 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.051005 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.051627 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.051631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.051635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.051743 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.052055 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.052058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.052060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.052367 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.052470 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.052765 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.052993 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":40}
I20260926 02:14:07.058292 3685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: Bootstrap starting.
I20260926 02:14:07.061950 3685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:07.063002 3685 log.cc:824] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:07.066093 3685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: No bootstrap required, opened a new log
I20260926 02:14:07.072942 3685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } }
I20260926 02:14:07.073391 3685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:07.073508 3685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40341a9abb464c30ae76521f28ee3be3, State: Initialized, Role: FOLLOWER
I20260926 02:14:07.074110 3685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } }
I20260926 02:14:07.074898 3691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } } }
I20260926 02:14:07.075712 3691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:14:07.076030 3685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:14:07.085575 3702 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:14:07.085709 3702 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:14:07.089599 3529 master_runner.cc:428] Detected that this master 40341a9abb464c30ae76521f28ee3be3 is joining an existing cluster
I20260926 02:14:07.089727 3529 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.152.188:45997
I20260926 02:14:07.113571 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.093992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.28.152.188:45997)
0926 02:14:07.094202 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.094208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.098389 (+ 4181us) server_negotiation.cc:407] Connection header received
0926 02:14:07.098565 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.098571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.098644 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.098731 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.100278 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.101552 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.103492 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.104134 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.104222 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.106809 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.106815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.106820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.106853 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.110444 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.111222 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.111231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.111234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.111410 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.111825 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.111829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.111831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.112173 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.112319 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.112846 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.113069 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:14:07.114671 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.094041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:45997 (local address 127.0.0.1:60298)
0926 02:14:07.098259 (+ 4218us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.098281 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.098453 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.098894 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.098899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.099678 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:07.100078 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.100096 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.101763 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.101769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.103332 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.103345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.104514 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.104520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.104677 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.105402 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:07.105441 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:07.106649 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:07.110570 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.110579 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.110584 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.111084 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.111553 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.111557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.111560 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.111708 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.112400 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:07.112403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:07.112534 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.113833 (+ 1299us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.114159 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4079,"mutex_wait_us":22,"thread_start_us":3944,"threads_started":1}
I20260926 02:14:07.118773 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.094024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46398 (local address 127.28.152.189:37903)
0926 02:14:07.094202 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.094208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.101894 (+ 7686us) server_negotiation.cc:407] Connection header received
0926 02:14:07.102161 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.102167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.102249 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.102317 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.103653 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.104656 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.106701 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.107805 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.107906 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.112578 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.112585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.112590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.112630 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.115720 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.116412 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.116417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.116421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.116596 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.117004 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.117009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.117012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.117364 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.117494 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.118205 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.118430 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 02:14:07.118848 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.097612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:37903 (local address 127.0.0.1:46398)
0926 02:14:07.101785 (+ 4173us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.101805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.102002 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.102501 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.102506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.103161 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:07.103500 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.103509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.104851 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.104855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.106517 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.106531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.108187 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.108195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.108392 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.108996 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:07.109019 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:07.112382 (+ 3363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:07.115886 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.115889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.115892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.116285 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.116712 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.116716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.116718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.116874 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.117470 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:07.117475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:07.117878 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.118071 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.118430 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3998,"mutex_wait_us":4,"thread_start_us":3507,"threads_started":1}
I20260926 02:14:07.127131 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.094522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.28.152.190:34891)
0926 02:14:07.094735 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.094740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.113718 (+ 18978us) server_negotiation.cc:407] Connection header received
0926 02:14:07.113958 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.113964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.114039 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.114094 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.115319 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.116346 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.118520 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.119191 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.119286 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.122096 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.122101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.122105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.122138 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.124619 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.125348 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.125353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.125355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.125473 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.125863 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.125866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.125868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.126133 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.126238 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.126726 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.126885 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 02:14:07.127380 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.097632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:43048)
0926 02:14:07.113596 (+ 15964us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.113613 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.113828 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.114273 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.114276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.114888 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:07.115150 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.115159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.116514 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.116520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.118348 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.118363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.119901 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.119908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.120093 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.120796 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:07.120818 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:07.121917 (+ 1099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:07.124749 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.124755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.124759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.125242 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.125573 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.125578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.125582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.125767 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.126238 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:07.126243 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:07.126459 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.126612 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.126939 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15828,"mutex_wait_us":53,"thread_start_us":15583,"threads_started":1}
I20260926 02:14:07.130247 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 3030
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388844, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.190@KRBTEST.COM for oryx/127.28.152.188@KRBTEST.COM
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.152.129@KRBTEST.COM
I20260926 02:14:07.147167 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.132713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:45997 (local address 127.0.0.1:60302)
0926 02:14:07.132935 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.132952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.133046 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.133543 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.133547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.133978 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:07.134117 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.134125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.135312 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.135318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.136687 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.136701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.137556 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.137563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.137720 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.138244 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:07.138280 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:07.140830 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:07.144469 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.144475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.144479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.145007 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.145446 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.145452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.145456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.145605 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.146119 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:07.146123 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:07.146292 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.146433 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.146720 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260926 02:14:07.147661 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.133093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.28.152.188:45997)
0926 02:14:07.133275 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.133279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.133291 (+ 12us) server_negotiation.cc:407] Connection header received
0926 02:14:07.133355 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.133358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.133411 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.133497 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.134243 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.135177 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.136834 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.137423 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.137528 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.141030 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.141042 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.141047 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.141086 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.144304 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.145171 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.145177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.145180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.145329 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.145704 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.145709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.145712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.145989 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.146117 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.146532 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.147289 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 02:14:07.149048 2969 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.152.188:45997: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "40341a9abb464c30ae76521f28ee3be3"
member_type: NON_VOTER
last_known_addr {
host: "127.28.152.188"
port: 45997
}
attrs {
promote: true
}
}
dest_uuid: "248a9ebcadfd4363957532d4e5a993ec"
cas_config_opid_index: 5
I20260926 02:14:07.150120 2969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } }
I20260926 02:14:07.151016 3711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.131916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.28.152.129:40489)
0926 02:14:07.132462 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.132471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.132502 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:14:07.132573 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.132582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.132882 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.133748 (+ 866us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.134631 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.135828 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.137153 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.138047 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.138179 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.143533 (+ 5354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.143551 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.143571 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.143615 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.147176 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.148480 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.148490 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.148494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.148718 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.149206 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.149211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.149214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.149655 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.149968 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.150373 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.150641 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":499,"threads_started":1}
I20260926 02:14:07.152462 3153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:07.152722 3658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:07.156893 3658 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 FOLLOWER]: Refusing update from remote peer 248a9ebcadfd4363957532d4e5a993ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 02:14:07.158304 3577 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 LEARNER]: Refusing update from remote peer 248a9ebcadfd4363957532d4e5a993ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 02:14:07.159082 3527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 40341a9abb464c30ae76521f28ee3be3 (127.28.152.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.160687 3712 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260926 02:14:07.174700 3577 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 40341a9abb464c30ae76521f28ee3be3 (127.28.152.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.180783 3605 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.181084 3605 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: This master's current role is: LEARNER
I20260926 02:14:07.182066 3716 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:07.182915 3527 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.152.188:45997
I20260926 02:14:07.183579 3529 master.cc:537] Master@127.28.152.188:45997 shutting down...
I20260926 02:14:07.190925 3712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.191340 3712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:07.196066 3529 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:07.196509 3529 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:07.196713 3529 tablet_replica.cc:330] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: stopping tablet replica
I20260926 02:14:07.199116 3527 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec: attempting to promote NON_VOTER f31a651ed7d94d5caa654cd28802f45e to VOTER
I20260926 02:14:07.200470 3527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } }
I20260926 02:14:07.205965 3716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: NON_VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: true } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.206372 3716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:07.210891 3529 master.cc:552] Master@127.28.152.188:45997 shutdown complete.
I20260926 02:14:07.211027 3529 master_runner.cc:305] Clearing existing system tablet
I20260926 02:14:07.211453 3529 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:07.211606 3529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:07.213747 3577 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 LEARNER]: Refusing update from remote peer 248a9ebcadfd4363957532d4e5a993ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 02:14:07.214493 3717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 02:14:07.217547 3717 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f31a651ed7d94d5caa654cd28802f45e (127.28.152.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.218315 3577 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f31a651ed7d94d5caa654cd28802f45e (127.28.152.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.221835 3605 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.222062 3605 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:14:07.223229 3716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.223511 3716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: This master's current role is: LEADER
W20260926 02:14:07.224109 2953 connection.cc:698] client connection to 127.28.152.188:45997 send error: Network error: sendmsg error: Broken pipe (error 32)
I20260926 02:14:07.224360 3716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.224643 3716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:07.226290 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.225754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:45997 (local address 127.0.0.1:60304)
0926 02:14:07.225929 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.226083 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 02:14:07.226682 2953 consensus_peers.cc:487] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec -> Peer 40341a9abb464c30ae76521f28ee3be3 (127.28.152.188:45997): Couldn't send request to peer 40341a9abb464c30ae76521f28ee3be3. Status: Network error: Client connection negotiation failed: client connection to 127.28.152.188:45997: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 02:14:07.237879 3529 fs_manager.cc:709] Time spent opening directory manager: real 0.025s user 0.017s sys 0.003s
I20260926 02:14:07.240231 3721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:07.240635 3529 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:07.240890 3529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 02:14:07.240993 3529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/wal
uuid: "40341a9abb464c30ae76521f28ee3be3"
format_stamp: "Formatted at 2026-09-26 02:14:06 on dist-test-slave-hllj"
server_key: "78cbd2f13841b9bdef8d7442e7ea644a"
server_key_iv: "cc2117f2219d2c1f4fef64b3b5f26145"
server_key_version: "encryptionkey@0"
I20260926 02:14:07.246379 3529 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:07.253659 3529 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 02:14:07.253883 3529 log.cc:1196] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 02:14:07.254387 3529 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: Deleting consensus metadata
I20260926 02:14:07.254635 3529 master_runner.cc:315] Copying system tablet from 127.28.152.190:34891
I20260926 02:14:07.257839 3529 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: tablet copy: Beginning tablet copy session from remote peer at address 127.28.152.190:34891
I20260926 02:14:07.275851 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.258733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.28.152.190:34891)
0926 02:14:07.258959 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.258966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.261733 (+ 2767us) server_negotiation.cc:407] Connection header received
0926 02:14:07.261926 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.261932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.262015 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.262125 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.263242 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.264428 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.266421 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.267024 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.267156 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.269352 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.269358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.269362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.269398 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.272756 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.273629 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.273635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.273639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.273798 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.274226 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.274231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.274234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.274606 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.274746 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.275313 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.275553 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 02:14:07.276688 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.258702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:43060)
0926 02:14:07.261580 (+ 2878us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.261609 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.261792 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.262165 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.262170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.262730 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:07.263006 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.263019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.264633 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.264638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.266267 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.266286 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.267199 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.267206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.267393 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.268067 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:07.268090 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:07.269152 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:07.272914 (+ 3762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.272923 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.272928 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.273492 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.273921 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.273927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.273931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.274108 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.274811 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:07.274816 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:07.275023 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.275169 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.275489 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2726,"thread_start_us":2583,"threads_started":1}
I20260926 02:14:07.277537 2989 tablet_copy_service.cc:140] P 248a9ebcadfd4363957532d4e5a993ec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 40341a9abb464c30ae76521f28ee3be3 ({username='oryx', principal='oryx/127.28.152.188@KRBTEST.COM'} at 127.0.0.1:43060)
I20260926 02:14:07.277698 2989 tablet_copy_service.cc:161] P 248a9ebcadfd4363957532d4e5a993ec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 40341a9abb464c30ae76521f28ee3be3 at {username='oryx', principal='oryx/127.28.152.188@KRBTEST.COM'} at 127.0.0.1:43060: session id = 40341a9abb464c30ae76521f28ee3be3-00000000000000000000000000000000
I20260926 02:14:07.281790 2989 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 02:14:07.283843 3529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 02:14:07.291181 3529 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: tablet copy: Starting download of 0 data blocks...
I20260926 02:14:07.291525 3529 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: tablet copy: Starting download of 1 WAL segments...
I20260926 02:14:07.293926 3529 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 02:14:07.298576 2989 tablet_copy_service.cc:342] P 248a9ebcadfd4363957532d4e5a993ec: Request end of tablet copy session 40341a9abb464c30ae76521f28ee3be3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.152.188@KRBTEST.COM'} at 127.0.0.1:43060
I20260926 02:14:07.298739 2989 tablet_copy_service.cc:433] P 248a9ebcadfd4363957532d4e5a993ec: ending tablet copy session 40341a9abb464c30ae76521f28ee3be3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 40341a9abb464c30ae76521f28ee3be3
I20260926 02:14:07.307157 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 3030
I20260926 02:14:07.311678 3529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:07.314852 3728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:07.315305 3729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:07.322625 3731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:07.323294 3529 server_base.cc:1034] running on GCE node
I20260926 02:14:07.323688 3529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:07.324414 3529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:07.325614 3529 hybrid_clock.cc:648] HybridClock initialized: now 1790388847325573 us; error 33 us; skew 500 ppm
I20260926 02:14:07.325948 3529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388847, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:07.327937 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:40489
--local_ip_for_outbound_sockets=127.28.152.129
--tserver_master_addrs=127.28.152.190:34891
--webserver_port=41233
--webserver_interface=127.28.152.129
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab
--principal=oryx/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.152.190:34891,127.28.152.189:37903
--tserver_master_addrs=127.28.152.190:34891,127.28.152.189:37903,127.28.152.188:45997 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:14:07.328416 3529 init.cc:377] Logged in from keytab as oryx/127.28.152.188@KRBTEST.COM (short username oryx)
I20260926 02:14:07.334400 3529 webserver.cc:460] Webserver started at http://127.28.152.188:34307/ using document root <none> and password file <none>
I20260926 02:14:07.334784 3529 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:07.334892 3529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:07.338104 3529 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260926 02:14:07.340186 3739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:07.340642 3529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 02:14:07.340782 3529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/wal
uuid: "40341a9abb464c30ae76521f28ee3be3"
format_stamp: "Formatted at 2026-09-26 02:14:06 on dist-test-slave-hllj"
server_key: "78cbd2f13841b9bdef8d7442e7ea644a"
server_key_iv: "cc2117f2219d2c1f4fef64b3b5f26145"
server_key_version: "encryptionkey@0"
I20260926 02:14:07.340970 3529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:07.356513 3529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:07.360396 3529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:07.407575 3529 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.188:45997
I20260926 02:14:07.407632 3790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.188:45997 every 8 connection(s)
I20260926 02:14:07.408655 3529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
I20260926 02:14:07.413805 3791 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:14:07.419464 3791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: Bootstrap starting.
I20260926 02:14:07.438555 3791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:14:07.439078 3791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: Bootstrap complete.
I20260926 02:14:07.440146 3791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } }
I20260926 02:14:07.440452 3791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40341a9abb464c30ae76521f28ee3be3, State: Initialized, Role: LEARNER
I20260926 02:14:07.440709 3791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } }
I20260926 02:14:07.441819 3791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:14:07.441581 3794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.442382 3794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: This master's current role is: LEARNER
I20260926 02:14:07.451627 3805 catalog_manager.cc:1259] Loaded cluster ID: aa430a297210411e86ecefbf19f28df6
I20260926 02:14:07.451794 3805 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: loading cluster ID for follower catalog manager: success
I20260926 02:14:07.455617 3805 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: acquiring CA information for follower catalog manager: success
I20260926 02:14:07.458241 3805 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:14:07.653036 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.646462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:45997 (local address 127.0.0.1:60312)
0926 02:14:07.646750 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.646771 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.646902 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.647712 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.647717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.647746 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:07.647940 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.647950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.649304 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.649308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.650493 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.650506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.651639 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.651659 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.652351 (+ 692us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.652379 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.652578 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.652756 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260926 02:14:07.653174 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.646640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.28.152.188:45997)
0926 02:14:07.647365 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.647369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.647384 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:14:07.647442 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.647445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.647521 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.647678 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:07.648086 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.649117 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.650647 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.651464 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.651639 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.651864 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.652595 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.652742 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":467,"threads_started":1}
I20260926 02:14:07.654134 3766 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 02:14:07.654857 3794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 248a9ebcadfd4363957532d4e5a993ec. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:07.655190 3794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: This master's current role is: LEARNER
W20260926 02:14:07.681694 3736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:07.682013 3736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:07.682152 3736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:07.690024 3736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:07.690140 3736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:07.690287 3736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:14:07.700203 3736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab
--principal=oryx/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:40489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=41233
--tserver_master_addrs=127.28.152.190:34891,127.28.152.189:37903,127.28.152.188:45997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:07.701572 3736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:07.702909 3736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:07.716202 3814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:07.716275 3812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:07.716238 3811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:07.717360 3736 server_base.cc:1034] running on GCE node
I20260926 02:14:07.717957 3736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:07.718901 3736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:07.720105 3736 hybrid_clock.cc:648] HybridClock initialized: now 1790388847720039 us; error 52 us; skew 500 ppm
I20260926 02:14:07.720431 3736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388847, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:07.723876 3736 init.cc:377] Logged in from keytab as oryx/127.28.152.129@KRBTEST.COM (short username oryx)
I20260926 02:14:07.725507 3736 webserver.cc:460] Webserver started at http://127.28.152.129:41233/ using document root <none> and password file <none>
I20260926 02:14:07.726235 3736 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:07.726367 3736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:07.731895 3736 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 02:14:07.734826 3821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:07.736227 3736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 02:14:07.736384 3736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "8e34eff83373477c97735ff40f839576"
format_stamp: "Formatted at 2026-09-26 02:14:04 on dist-test-slave-hllj"
server_key: "b51c46bda6f4412ec4407bbdc3f751af"
server_key_iv: "f30eea47cbe2a14a469102dffa41ae20"
server_key_version: "encryptionkey@0"
I20260926 02:14:07.737370 3736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:07.759892 3736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:07.764479 3736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:07.764741 3736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:07.765831 3736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:07.767316 3736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:07.767417 3736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:07.767531 3736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:07.767609 3736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:07.804418 3736 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:40489
I20260926 02:14:07.804474 3933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:40489 every 8 connection(s)
I20260926 02:14:07.805950 3736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:14:07.812449 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 3736
I20260926 02:14:07.812784 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 3164
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388847, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.189@KRBTEST.COM
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388847, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.188@KRBTEST.COM
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388847, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
Sep 26 02:14:07 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.152.130@KRBTEST.COM
I20260926 02:14:07.846138 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.809683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:43403 (local address 127.28.152.188:45997)
0926 02:14:07.809898 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.809906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.822122 (+ 12216us) server_negotiation.cc:407] Connection header received
0926 02:14:07.824381 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.824388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.824468 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.824531 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.827480 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.828534 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.831904 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.832538 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.832640 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.838510 (+ 5870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.838517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.838522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.838563 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.841925 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.843085 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.843091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.843094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.843244 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.843803 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.843808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.843811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.844223 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.844358 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.845489 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.845729 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:14:07.847805 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.809564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:50093 (local address 127.28.152.189:37903)
0926 02:14:07.819328 (+ 9764us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.819334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.821592 (+ 2258us) server_negotiation.cc:407] Connection header received
0926 02:14:07.827462 (+ 5870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.827468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.827569 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.827668 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.829467 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.830211 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.831963 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.832556 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.832690 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.838593 (+ 5903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.838602 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.838607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.838648 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.842775 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.843758 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.843764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.843768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.843913 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.844366 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.844371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.844374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.844730 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.844870 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.847213 (+ 2343us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.847450 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9627,"thread_start_us":11011,"threads_started":1}
I20260926 02:14:07.849323 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.813485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:45997 (local address 127.28.152.129:43403)
0926 02:14:07.821999 (+ 8514us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.822017 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.824222 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.824996 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.825010 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.826058 (+ 1048us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:07.827106 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.827140 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.828796 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.828802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.831698 (+ 2896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.831718 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.833093 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.833101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.833456 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.834366 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:07.834410 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:07.838206 (+ 3796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:07.842110 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.842169 (+ 59us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.842191 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.842942 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.843393 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.843398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.843401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.843670 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.844452 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:07.844461 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:07.844960 (+ 499us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.846325 (+ 1365us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.846733 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8388,"mutex_wait_us":174,"spinlock_wait_cycles":24064,"thread_start_us":7657,"threads_started":1}
I20260926 02:14:07.849349 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.811593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:37903 (local address 127.28.152.129:50093)
0926 02:14:07.821468 (+ 9875us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.821492 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.827328 (+ 5836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.827905 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.827910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.828429 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:07.829285 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.829299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.830369 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.830374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.831802 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.831812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.833000 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.833007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.833447 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.834413 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:07.834437 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:07.838207 (+ 3770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:07.842925 (+ 4718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.842930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.842934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.843606 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.844041 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.844046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.844049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.844208 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.844931 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:07.844936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:07.845113 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.845952 (+ 839us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.846588 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9712,"mutex_wait_us":91,"spinlock_wait_cycles":37504,"thread_start_us":7345,"threads_started":1}
I20260926 02:14:07.851035 3935 heartbeater.cc:347] Connected to a master server at 127.28.152.188:45997
I20260926 02:14:07.851099 3936 heartbeater.cc:347] Connected to a master server at 127.28.152.189:37903
I20260926 02:14:07.851450 3936 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:07.851456 3935 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:07.852381 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.810961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.129:41963)
0926 02:14:07.819479 (+ 8518us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:07.819566 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:07.823808 (+ 4242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:07.830927 (+ 7119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:07.830932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:07.831435 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:07.831790 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.831804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.833212 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.833218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.835146 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:07.835166 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.836380 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.836388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:07.836567 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.838007 (+ 1440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:07.838032 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:07.842534 (+ 4502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:07.847651 (+ 5117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.847654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.847657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.848496 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.848957 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:07.848961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:07.848965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:07.849189 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:07.850726 (+ 1537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:07.850731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:07.850906 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 02:14:07.851580 (+ 674us) client_negotiation.cc:240] Negotiation successful
0926 02:14:07.851903 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8128,"mutex_wait_us":45,"thread_start_us":9302,"threads_started":1}
I20260926 02:14:07.852517 3936 heartbeater.cc:511] Master 127.28.152.189:37903 requested a full tablet report, sending...
I20260926 02:14:07.853330 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.809254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:41963 (local address 127.28.152.190:34891)
0926 02:14:07.830600 (+ 21346us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.830606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.830625 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:14:07.830681 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.830686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.830767 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.830873 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.831975 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.833013 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.835334 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.835968 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.836104 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.842791 (+ 6687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.842846 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.842854 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.842930 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.847483 (+ 4553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.848669 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.848675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.848678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.848827 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.849327 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.849332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.849334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.850488 (+ 1154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.850670 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.852721 (+ 2051us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.852954 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21215,"thread_start_us":23650,"threads_started":1}
I20260926 02:14:07.853399 3935 heartbeater.cc:511] Master 127.28.152.188:45997 requested a full tablet report, sending...
I20260926 02:14:07.854411 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:07.815365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52452 (local address 127.28.152.130:34051)
0926 02:14:07.829530 (+ 14165us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:07.829539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:07.829569 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:14:07.829639 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:07.829647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:07.829922 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:07.830089 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:07.833063 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.834513 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.836461 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:07.837712 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:07.837858 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:07.846318 (+ 8460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:07.846336 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:07.846356 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:07.846398 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:07.850012 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.851349 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.851365 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.851369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.851623 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:07.852138 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:07.852143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:07.852146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:07.852727 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:07.852999 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:07.853834 (+ 835us) server_negotiation.cc:299] Negotiation successful
0926 02:14:07.854109 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13978,"thread_start_us":13828,"threads_started":1}
I20260926 02:14:07.854867 3567 ts_manager.cc:194] Registered new tserver with Master: 8e34eff83373477c97735ff40f839576 (127.28.152.129:40489)
I20260926 02:14:07.855551 3934 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34891
I20260926 02:14:07.855688 3934 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:07.856133 3934 heartbeater.cc:511] Master 127.28.152.190:34891 requested a full tablet report, sending...
I20260926 02:14:07.856863 3756 ts_manager.cc:194] Registered new tserver with Master: 8e34eff83373477c97735ff40f839576 (127.28.152.129:40489)
I20260926 02:14:07.857355 2969 ts_manager.cc:194] Re-registered known tserver with Master: 8e34eff83373477c97735ff40f839576 (127.28.152.129:40489)
I20260926 02:14:07.857810 3287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:07.858340 2969 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:41963
I20260926 02:14:07.940661 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 3164
I20260926 02:14:07.954674 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:34051
--local_ip_for_outbound_sockets=127.28.152.130
--tserver_master_addrs=127.28.152.190:34891
--webserver_port=41601
--webserver_interface=127.28.152.130
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab
--principal=oryx/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.152.190:34891,127.28.152.189:37903
--tserver_master_addrs=127.28.152.190:34891,127.28.152.189:37903,127.28.152.188:45997 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:14:08.145973 3716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: NON_VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: true } } }
I20260926 02:14:08.146378 3716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:08.148069 3716 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec: attempting to promote NON_VOTER 40341a9abb464c30ae76521f28ee3be3 to VOTER
I20260926 02:14:08.148867 3716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } }
I20260926 02:14:08.150882 3577 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 FOLLOWER]: Refusing update from remote peer 248a9ebcadfd4363957532d4e5a993ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 02:14:08.150938 3766 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 LEARNER]: Refusing update from remote peer 248a9ebcadfd4363957532d4e5a993ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 02:14:08.151533 3947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 02:14:08.151926 3948 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 02:14:08.154999 3947 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 40341a9abb464c30ae76521f28ee3be3 (127.28.152.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:08.155834 3577 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 40341a9abb464c30ae76521f28ee3be3 (127.28.152.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:08.159900 3947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:08.160247 3947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:08.160557 3947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:08.160835 3947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:08.161419 3951 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:08.162093 3951 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:14:08.170029 3948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:08.170364 3948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:08.170552 3766 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 40341a9abb464c30ae76521f28ee3be3 (127.28.152.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:08.173728 3952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "248a9ebcadfd4363957532d4e5a993ec" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:08.174081 3952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 02:14:08.287694 3946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:08.287999 3946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:08.288121 3946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:08.295521 3946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:08.295646 3946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:08.295759 3946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:14:08.305581 3946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab
--principal=oryx/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:34051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=41601
--tserver_master_addrs=127.28.152.190:34891,127.28.152.189:37903,127.28.152.188:45997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:08.306860 3946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:08.308138 3946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:08.321161 3957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:08.321274 3958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:08.321343 3960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:08.322783 3946 server_base.cc:1034] running on GCE node
I20260926 02:14:08.323557 3946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:08.324800 3946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:08.326018 3946 hybrid_clock.cc:648] HybridClock initialized: now 1790388848325926 us; error 76 us; skew 500 ppm
I20260926 02:14:08.326380 3946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:08 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388848, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:08.330513 3946 init.cc:377] Logged in from keytab as oryx/127.28.152.130@KRBTEST.COM (short username oryx)
I20260926 02:14:08.332090 3946 webserver.cc:460] Webserver started at http://127.28.152.130:41601/ using document root <none> and password file <none>
I20260926 02:14:08.332801 3946 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:08.332934 3946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:08.338553 3946 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 02:14:08.341403 3967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:08.343119 3946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:14:08.343286 3946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "c49afa0c59cd4109bd029349db882ecd"
format_stamp: "Formatted at 2026-09-26 02:14:05 on dist-test-slave-hllj"
server_key: "119793e138150ad9049d31c8c6bb3d5d"
server_key_iv: "0838d5b54613b0cb1c640f872351d244"
server_key_version: "encryptionkey@0"
I20260926 02:14:08.344347 3946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:08.370028 3946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:08.374614 3946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:08.374873 3946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:08.375875 3946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:08.377398 3946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:08.377488 3946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:08.377576 3946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:08.377641 3946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:08.407522 3946 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:34051
I20260926 02:14:08.407582 4079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:34051 every 8 connection(s)
I20260926 02:14:08.408936 3946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:14:08.414702 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 3946
I20260926 02:14:08.414927 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 3297
Sep 26 02:14:08 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.152.131@KRBTEST.COM
Sep 26 02:14:08 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388848, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.189@KRBTEST.COM
Sep 26 02:14:08 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388848, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
Sep 26 02:14:08 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388848, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.188@KRBTEST.COM
I20260926 02:14:08.456429 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:08.415839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.28.152.131:36667)
0926 02:14:08.431266 (+ 15427us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:08.431278 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:08.434924 (+ 3646us) server_negotiation.cc:407] Connection header received
0926 02:14:08.435091 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:08.435101 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:08.435365 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:08.435515 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:08.437148 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.438207 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.439646 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.440224 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.440358 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:08.447246 (+ 6888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:08.447267 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:08.447296 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:08.447343 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:08.451289 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:08.452177 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:08.452188 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:08.452191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:08.452371 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:08.452840 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:08.452846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:08.452849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:08.453321 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:08.453620 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:08.454971 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0926 02:14:08.455244 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15181,"thread_start_us":15465,"threads_started":1}
I20260926 02:14:08.457697 3420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:08.463287 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:08.411724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:59991 (local address 127.28.152.190:34891)
0926 02:14:08.412666 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:08.412672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:08.438767 (+ 26095us) server_negotiation.cc:407] Connection header received
0926 02:14:08.443108 (+ 4341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:08.443114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:08.443218 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:08.443330 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:08.445981 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.447076 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.449883 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.450310 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.450431 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:08.455660 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:08.455665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:08.455668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:08.455712 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:08.458970 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:08.460030 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:08.460035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:08.460037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:08.460183 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:08.460681 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:08.460685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:08.460687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:08.460962 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:08.461071 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:08.462658 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0926 02:14:08.462897 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":678,"threads_started":1}
I20260926 02:14:08.464753 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:08.412476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.130:59991)
0926 02:14:08.438545 (+ 26069us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:08.438617 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:08.442761 (+ 4144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:08.443641 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:08.443654 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:08.444593 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:08.445761 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:08.445796 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.447322 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.447327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:08.449744 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:08.449758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.450534 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.450541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:08.450894 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:08.452250 (+ 1356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:08.452271 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:08.455465 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:08.459160 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:08.459172 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:08.459193 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:08.459884 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:08.460288 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:08.460293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:08.460297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:08.460563 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:08.461072 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:08.461081 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:08.461639 (+ 558us) client_negotiation.cc:769] Sending connection context
0926 02:14:08.461953 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 02:14:08.462582 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25656,"mutex_wait_us":302,"spinlock_wait_cycles":47872,"thread_start_us":29818,"threads_started":1}
I20260926 02:14:08.464977 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:08.413015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:45093 (local address 127.28.152.189:37903)
0926 02:14:08.437705 (+ 24690us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:08.437712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:08.441738 (+ 4026us) server_negotiation.cc:407] Connection header received
0926 02:14:08.443155 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:08.443161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:08.443228 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:08.443307 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:08.446018 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.446822 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.449927 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.450548 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.450647 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:08.455436 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:08.455444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:08.455448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:08.455503 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:08.458960 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:08.461685 (+ 2725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:08.461691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:08.461695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:08.461859 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:08.462271 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:08.462276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:08.462279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:08.462904 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:08.463043 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:08.463647 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 02:14:08.463875 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24543,"thread_start_us":24716,"threads_started":1}
I20260926 02:14:08.465947 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:08.413000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:37903 (local address 127.28.152.130:45093)
0926 02:14:08.441547 (+ 28547us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:08.441568 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:08.442763 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:08.443641 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:08.443654 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:08.444600 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:08.445808 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:08.445819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.447003 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.447006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:08.449740 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:08.449756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.450952 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.450959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:08.451173 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:08.451691 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:08.451713 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:08.455152 (+ 3439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:08.460038 (+ 4886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:08.460043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:08.460048 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:08.461557 (+ 1509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:08.461995 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:08.461999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:08.462003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:08.462152 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:08.463120 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:08.463125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:08.463277 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 02:14:08.465238 (+ 1961us) client_negotiation.cc:240] Negotiation successful
0926 02:14:08.465542 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28401,"mutex_wait_us":72,"spinlock_wait_cycles":29824,"thread_start_us":29304,"threads_started":1}
I20260926 02:14:08.467306 4082 heartbeater.cc:347] Connected to a master server at 127.28.152.189:37903
I20260926 02:14:08.467499 4080 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34891
I20260926 02:14:08.467640 4082 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:08.467639 4080 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:08.468616 4082 heartbeater.cc:511] Master 127.28.152.189:37903 requested a full tablet report, sending...
I20260926 02:14:08.469493 4080 heartbeater.cc:511] Master 127.28.152.190:34891 requested a full tablet report, sending...
I20260926 02:14:08.470330 3567 ts_manager.cc:194] Registered new tserver with Master: c49afa0c59cd4109bd029349db882ecd (127.28.152.130:34051)
I20260926 02:14:08.470618 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:08.413466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:45997 (local address 127.28.152.130:37367)
0926 02:14:08.442223 (+ 28757us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:08.442240 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:08.442819 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:08.443925 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:08.443929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:08.444704 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:08.445759 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:08.445792 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.448906 (+ 3114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.448911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:08.451854 (+ 2943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:08.451868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.453562 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.453567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:08.453688 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:08.454258 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:08.454293 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:08.461227 (+ 6934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:08.467459 (+ 6232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:08.467464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:08.467468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:08.468224 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:08.468657 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:08.468661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:08.468663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:08.468822 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:08.469517 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:08.469522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:08.469745 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 02:14:08.469890 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:14:08.470171 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28649,"mutex_wait_us":90,"thread_start_us":27836,"threads_started":1}
I20260926 02:14:08.470647 2969 ts_manager.cc:194] Re-registered known tserver with Master: c49afa0c59cd4109bd029349db882ecd (127.28.152.130:34051)
I20260926 02:14:08.470857 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:08.413344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:37367 (local address 127.28.152.188:45997)
0926 02:14:08.413815 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:08.413820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:08.442331 (+ 28511us) server_negotiation.cc:407] Connection header received
0926 02:14:08.443537 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:08.443543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:08.443635 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:08.443711 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:08.447268 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.448742 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.452020 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:08.453206 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:08.453346 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:08.461795 (+ 8449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:08.461801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:08.461806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:08.461845 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:08.466968 (+ 5123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:08.468380 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:08.468386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:08.468389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:08.468542 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:08.468935 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:08.468941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:08.468945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:08.469313 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:08.469458 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:08.470286 (+ 828us) server_negotiation.cc:299] Negotiation successful
0926 02:14:08.470506 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":220,"threads_started":1}
I20260926 02:14:08.471596 4081 heartbeater.cc:347] Connected to a master server at 127.28.152.188:45997
I20260926 02:14:08.471638 2969 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:59991
I20260926 02:14:08.471735 4081 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:08.472172 4081 heartbeater.cc:511] Master 127.28.152.188:45997 requested a full tablet report, sending...
I20260926 02:14:08.473351 3756 ts_manager.cc:194] Registered new tserver with Master: c49afa0c59cd4109bd029349db882ecd (127.28.152.130:34051)
I20260926 02:14:08.552538 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 3297
I20260926 02:14:08.566169 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:36667
--local_ip_for_outbound_sockets=127.28.152.131
--tserver_master_addrs=127.28.152.190:34891
--webserver_port=38947
--webserver_interface=127.28.152.131
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab
--principal=oryx/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.152.190:34891,127.28.152.189:37903
--tserver_master_addrs=127.28.152.190:34891,127.28.152.189:37903,127.28.152.188:45997 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:14:08.861172 3934 heartbeater.cc:503] Master 127.28.152.190:34891 was elected leader, sending a full tablet report...
W20260926 02:14:08.907765 4093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:08.908072 4093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:08.908226 4093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:08.916354 4093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:08.916460 4093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:08.916611 4093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:14:08.926714 4093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab
--principal=oryx/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:36667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=38947
--tserver_master_addrs=127.28.152.190:34891,127.28.152.189:37903,127.28.152.188:45997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:08.928012 4093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:08.929440 4093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:08.943002 4098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:08.943032 4101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:08.942965 4099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:08.944002 4093 server_base.cc:1034] running on GCE node
I20260926 02:14:08.944664 4093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:08.945856 4093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:08.947078 4093 hybrid_clock.cc:648] HybridClock initialized: now 1790388848946975 us; error 104 us; skew 500 ppm
I20260926 02:14:08.947420 4093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:08 dist-test-slave-hllj krb5kdc[2924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388848, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:08.950802 4093 init.cc:377] Logged in from keytab as oryx/127.28.152.131@KRBTEST.COM (short username oryx)
I20260926 02:14:08.952353 4093 webserver.cc:460] Webserver started at http://127.28.152.131:38947/ using document root <none> and password file <none>
I20260926 02:14:08.953042 4093 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:08.953200 4093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:08.958791 4093 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:14:08.961661 4108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:08.963084 4093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:14:08.963241 4093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "320e0037c6754a51b79146fc5f118f45"
format_stamp: "Formatted at 2026-09-26 02:14:05 on dist-test-slave-hllj"
server_key: "f33e6c6efe82231e46ed41bd18b3e46a"
server_key_iv: "c22dd69c836432051591d6753b6c25e9"
server_key_version: "encryptionkey@0"
I20260926 02:14:08.964224 4093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:08.993376 4093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:08.997916 4093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:08.998199 4093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:08.999183 4093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:09.000593 4093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:09.000694 4093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:09.000802 4093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:09.000869 4093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:09.031675 4093 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:36667
I20260926 02:14:09.031718 4220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:36667 every 8 connection(s)
I20260926 02:14:09.033109 4093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:14:09.037984 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 4093
Sep 26 02:14:09 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388848, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.189@KRBTEST.COM
Sep 26 02:14:09 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388848, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.188@KRBTEST.COM
I20260926 02:14:09.075697 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.036327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:40165 (local address 127.28.152.189:37903)
0926 02:14:09.049397 (+ 13070us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.049404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.049430 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:14:09.052197 (+ 2767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.052202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.052302 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.052400 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:09.055055 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.056134 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.058882 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.059521 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.059618 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.067808 (+ 8190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:09.067815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:09.067820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:09.067865 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:09.071594 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.072649 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.072656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.072660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.072832 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.073385 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.073390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.073394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.073718 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:09.073876 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.074977 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.075252 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12931,"thread_start_us":13117,"threads_started":1}
I20260926 02:14:09.078460 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.036986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:37903 (local address 127.28.152.131:40165)
0926 02:14:09.045651 (+ 8665us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:09.045720 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:09.051875 (+ 6155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:09.052703 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:09.052716 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:09.053771 (+ 1055us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:09.054829 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.054859 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.056348 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.056353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.058718 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.058732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.059966 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.059974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.060300 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.061374 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:09.061407 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:09.065762 (+ 4355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:09.071798 (+ 6036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:09.071808 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:09.071829 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:09.072506 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:09.072972 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:09.072977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:09.072981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:09.073251 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:09.073976 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:09.073984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:09.074490 (+ 506us) client_negotiation.cc:769] Sending connection context
0926 02:14:09.076036 (+ 1546us) client_negotiation.cc:240] Negotiation successful
0926 02:14:09.076741 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8333,"mutex_wait_us":52,"spinlock_wait_cycles":1920,"thread_start_us":16466,"threads_started":1}
I20260926 02:14:09.080693 4223 heartbeater.cc:347] Connected to a master server at 127.28.152.189:37903
I20260926 02:14:09.081035 4223 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:09.081794 4223 heartbeater.cc:511] Master 127.28.152.189:37903 requested a full tablet report, sending...
Sep 26 02:14:09 dist-test-slave-hllj krb5kdc[2924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388848, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:09.083658 3567 ts_manager.cc:194] Registered new tserver with Master: 320e0037c6754a51b79146fc5f118f45 (127.28.152.131:36667)
I20260926 02:14:09.085923 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.036175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:51853 (local address 127.28.152.188:45997)
0926 02:14:09.049855 (+ 13680us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.049859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.054002 (+ 4143us) server_negotiation.cc:407] Connection header received
0926 02:14:09.054228 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.054234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.054320 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.054393 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:09.055688 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.056763 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.058896 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.059521 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.059618 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.066808 (+ 7190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:09.066815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:09.066820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:09.066864 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:09.070509 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.083267 (+ 12758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.083275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.083279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.083577 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.084074 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.084081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.084085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.084471 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:09.084613 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.085255 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.085538 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13567,"thread_start_us":13477,"threads_started":1}
I20260926 02:14:09.086988 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.040157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:45997 (local address 127.28.152.131:51853)
0926 02:14:09.053896 (+ 13739us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:09.053920 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:09.054099 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:09.054609 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:09.054613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:09.055257 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:09.055524 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.055536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.056974 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.056979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.058718 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.058732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.060764 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.060773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.060948 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.063625 (+ 2677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:09.063649 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:09.066555 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:09.080482 (+ 13927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:09.080488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:09.080493 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:09.081077 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:09.083739 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:09.083745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:09.083750 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:09.083947 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:09.084691 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:09.084699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:09.084881 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 02:14:09.086202 (+ 1321us) client_negotiation.cc:240] Negotiation successful
0926 02:14:09.086556 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13595,"mutex_wait_us":24,"thread_start_us":6420,"threads_started":1}
I20260926 02:14:09.088791 4222 heartbeater.cc:347] Connected to a master server at 127.28.152.188:45997
I20260926 02:14:09.088958 4222 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:09.089468 4222 heartbeater.cc:511] Master 127.28.152.188:45997 requested a full tablet report, sending...
I20260926 02:14:09.090737 3756 ts_manager.cc:194] Registered new tserver with Master: 320e0037c6754a51b79146fc5f118f45 (127.28.152.131:36667)
I20260926 02:14:09.093518 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.039770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:55991 (local address 127.28.152.190:34891)
0926 02:14:09.047781 (+ 8011us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.047788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.047825 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:14:09.052325 (+ 4500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.052332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.052420 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.052519 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:09.055118 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.056314 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.063350 (+ 7036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.064029 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.064130 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.085448 (+ 21318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:09.085456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:09.085461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:09.085503 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:09.089771 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.090582 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.090588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.090591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.090766 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.091863 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.091870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.091873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.092223 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:09.092363 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.092914 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.093163 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7820,"thread_start_us":6569,"threads_started":1}
I20260926 02:14:09.094673 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.036986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.131:55991)
0926 02:14:09.046894 (+ 9908us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:09.046917 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:09.051861 (+ 4944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:09.052944 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:09.052947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:09.054358 (+ 1411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:09.054827 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.054862 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.056542 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.056548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.062714 (+ 6166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.062736 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.064499 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.064506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.064703 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.076323 (+ 11620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:09.076347 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:09.085198 (+ 8851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:09.089941 (+ 4743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:09.089946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:09.089950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:09.090438 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:09.090997 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:09.091002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:09.091006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:09.091718 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:09.092441 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:09.092446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:09.092613 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 02:14:09.093959 (+ 1346us) client_negotiation.cc:240] Negotiation successful
0926 02:14:09.094265 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9731,"mutex_wait_us":492,"spinlock_wait_cycles":91008,"thread_start_us":8023,"threads_started":1}
I20260926 02:14:09.095894 4221 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34891
I20260926 02:14:09.096035 4221 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:09.096443 4221 heartbeater.cc:511] Master 127.28.152.190:34891 requested a full tablet report, sending...
I20260926 02:14:09.098191 2969 ts_manager.cc:194] Re-registered known tserver with Master: 320e0037c6754a51b79146fc5f118f45 (127.28.152.131:36667)
I20260926 02:14:09.099380 2969 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:55991
I20260926 02:14:09.102880 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.059263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.28.152.190:34891)
0926 02:14:09.082664 (+ 23401us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.082670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.082689 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:14:09.082750 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.082756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.082844 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.087199 (+ 4355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:09.087454 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.088705 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.090652 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.091263 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.092754 (+ 1491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.095478 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:09.095485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:09.095490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:09.095530 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:09.099251 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.100493 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.100499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.100503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.100655 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.101096 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.101101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.101105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.101516 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:09.101755 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.102265 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.102492 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23259,"thread_start_us":23764,"threads_started":1}
I20260926 02:14:09.108268 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.059618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60316 (local address 127.28.152.188:45997)
0926 02:14:09.084653 (+ 25035us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.084659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.089725 (+ 5066us) server_negotiation.cc:407] Connection header received
0926 02:14:09.089785 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.089790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.089876 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.089966 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:09.093011 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.094122 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.096078 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.096696 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.096834 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.101658 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:09.101665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:09.101669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:09.101711 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:09.105029 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.105875 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.105880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.105884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.106005 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.106476 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.106481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.106484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.106813 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:09.107019 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.107342 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.107542 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24913,"thread_start_us":24756,"threads_started":1}
I20260926 02:14:09.109239 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.065421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.28.152.189:37903)
0926 02:14:09.075956 (+ 10535us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.075973 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.092369 (+ 16396us) server_negotiation.cc:407] Connection header received
0926 02:14:09.092521 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.092526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.092627 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.092716 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:09.094967 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.096014 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.098075 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.098721 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.098834 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.102207 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:09.102214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:09.102219 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:09.102258 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:09.105503 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.106425 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.106431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.106434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.106586 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.107006 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.107012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.107015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.107343 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:09.107561 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.108625 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.108868 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10430,"thread_start_us":16500,"threads_started":1}
I20260926 02:14:09.106788 2969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:14:09.115672 2969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:14:09.145380 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.135977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.28.152.129:40489)
0926 02:14:09.137680 (+ 1703us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.137688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.138031 (+ 343us) server_negotiation.cc:407] Connection header received
0926 02:14:09.138329 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.138337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.138592 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.138787 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:09.139472 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.140709 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.143251 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.144326 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.144499 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.144698 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.144850 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.145019 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1533,"thread_start_us":1415,"threads_started":1}
I20260926 02:14:09.145473 4249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.135762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:40489 (local address 127.0.0.1:32936)
0926 02:14:09.137888 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:09.137914 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:09.138131 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:09.138982 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:09.138986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:09.139008 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:09.139311 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.139319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.140815 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.140821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.143109 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.143124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.144511 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.144517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.144688 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.144721 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:14:09.144807 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 02:14:09.145073 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2006,"thread_start_us":572,"threads_started":1}
I20260926 02:14:09.147854 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.135560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.28.152.130:34051)
0926 02:14:09.136107 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.136117 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.136738 (+ 621us) server_negotiation.cc:407] Connection header received
0926 02:14:09.137036 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.137045 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.137377 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.137557 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:09.139114 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.141411 (+ 2297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.144045 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.146050 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.146275 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.146507 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.147276 (+ 769us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.147543 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":317,"threads_started":1}
I20260926 02:14:09.148658 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.135329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:34051 (local address 127.0.0.1:52454)
0926 02:14:09.136622 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:09.136642 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:09.136849 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:09.137805 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:09.137808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:09.137828 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:09.138817 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.138832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.141627 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.141633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.143851 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.143867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.146720 (+ 2853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.146726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.146998 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.147036 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:14:09.148057 (+ 1021us) client_negotiation.cc:240] Negotiation successful
0926 02:14:09.148235 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"thread_start_us":2293,"threads_started":1}
I20260926 02:14:09.149683 3869 tablet_service.cc:1467] Processing CreateTablet for tablet 380a03b192004d9ca4f0d31d676f6206 (DEFAULT_TABLE table=test-table [id=e7437e1780a24db1a3b631db1daa4c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:09.151253 3869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380a03b192004d9ca4f0d31d676f6206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:09.152904 4015 tablet_service.cc:1467] Processing CreateTablet for tablet 380a03b192004d9ca4f0d31d676f6206 (DEFAULT_TABLE table=test-table [id=e7437e1780a24db1a3b631db1daa4c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:09.154413 4015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380a03b192004d9ca4f0d31d676f6206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:09.161327 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.137945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:36667 (local address 127.0.0.1:45576)
0926 02:14:09.138470 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:09.138490 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:09.138955 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:09.155454 (+ 16499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:09.155460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:09.155491 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:09.156340 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.156352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.157802 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.157808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.159121 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.159137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.160472 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.160479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.160668 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.160695 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 02:14:09.160781 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 02:14:09.160922 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":294,"threads_started":1}
I20260926 02:14:09.161326 4254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.138197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.28.152.131:36667)
0926 02:14:09.154900 (+ 16703us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.154906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.154937 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:14:09.155024 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.155032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.155237 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.155454 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:09.156537 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.157653 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.159294 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.160256 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.160422 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.160610 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.160843 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.161003 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16570,"thread_start_us":16325,"threads_started":1}
I20260926 02:14:09.165939 4156 tablet_service.cc:1467] Processing CreateTablet for tablet 380a03b192004d9ca4f0d31d676f6206 (DEFAULT_TABLE table=test-table [id=e7437e1780a24db1a3b631db1daa4c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:09.167202 4156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380a03b192004d9ca4f0d31d676f6206. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:09.170754 4256 tablet_bootstrap.cc:492] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: Bootstrap starting.
I20260926 02:14:09.173830 4255 tablet_bootstrap.cc:492] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: Bootstrap starting.
I20260926 02:14:09.174438 4256 tablet_bootstrap.cc:654] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:09.175853 4256 log.cc:824] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:09.177558 4255 tablet_bootstrap.cc:654] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:09.178627 4255 log.cc:824] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:09.179826 4256 tablet_bootstrap.cc:492] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: No bootstrap required, opened a new log
I20260926 02:14:09.180120 4256 ts_tablet_manager.cc:1397] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260926 02:14:09.181173 4255 tablet_bootstrap.cc:492] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: No bootstrap required, opened a new log
I20260926 02:14:09.181423 4255 ts_tablet_manager.cc:1397] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260926 02:14:09.183461 4259 tablet_bootstrap.cc:492] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: Bootstrap starting.
I20260926 02:14:09.187251 4259 tablet_bootstrap.cc:654] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:09.188544 4259 log.cc:824] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:09.188807 4256 raft_consensus.cc:348] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } } peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } }
I20260926 02:14:09.189188 4256 raft_consensus.cc:374] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:09.189284 4256 raft_consensus.cc:729] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c49afa0c59cd4109bd029349db882ecd, State: Initialized, Role: FOLLOWER
I20260926 02:14:09.190016 4256 consensus_queue.cc:260] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } } peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } }
I20260926 02:14:09.191115 4080 heartbeater.cc:503] Master 127.28.152.190:34891 was elected leader, sending a full tablet report...
I20260926 02:14:09.191179 4255 raft_consensus.cc:348] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } } peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } }
I20260926 02:14:09.191586 4255 raft_consensus.cc:374] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:09.191672 4255 raft_consensus.cc:729] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e34eff83373477c97735ff40f839576, State: Initialized, Role: FOLLOWER
I20260926 02:14:09.191763 4256 ts_tablet_manager.cc:1428] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260926 02:14:09.192312 4259 tablet_bootstrap.cc:492] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: No bootstrap required, opened a new log
I20260926 02:14:09.192576 4259 ts_tablet_manager.cc:1397] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260926 02:14:09.192335 4255 consensus_queue.cc:260] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } } peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } }
I20260926 02:14:09.197481 4255 ts_tablet_manager.cc:1428] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: Time spent starting tablet: real 0.016s user 0.004s sys 0.012s
I20260926 02:14:09.205017 4259 raft_consensus.cc:348] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } } peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } }
I20260926 02:14:09.205466 4259 raft_consensus.cc:374] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:09.205565 4259 raft_consensus.cc:729] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320e0037c6754a51b79146fc5f118f45, State: Initialized, Role: FOLLOWER
I20260926 02:14:09.206189 4259 consensus_queue.cc:260] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } } peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } }
I20260926 02:14:09.207293 4221 heartbeater.cc:503] Master 127.28.152.190:34891 was elected leader, sending a full tablet report...
I20260926 02:14:09.207904 4259 ts_tablet_manager.cc:1428] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
W20260926 02:14:09.300974 4224 tablet.cc:2369] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:14:09.315753 3937 tablet.cc:2369] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:14:09.414453 4083 tablet.cc:2369] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:14:09.495379 4261 raft_consensus.cc:482] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:14:09.495697 4261 raft_consensus.cc:504] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } } peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } }
I20260926 02:14:09.497424 4261 leader_election.cc:288] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e34eff83373477c97735ff40f839576 (127.28.152.129:40489), 320e0037c6754a51b79146fc5f118f45 (127.28.152.131:36667)
I20260926 02:14:09.503958 4254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.497973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:36003 (local address 127.28.152.131:36667)
0926 02:14:09.498167 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.498174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.498756 (+ 582us) server_negotiation.cc:407] Connection header received
0926 02:14:09.498928 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.498933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.499012 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.499078 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:09.499726 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.500616 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.502378 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.503133 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.503278 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.503365 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.503611 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.503741 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:14:09.504122 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.497679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:36667 (local address 127.28.152.130:36003)
0926 02:14:09.498635 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:09.498653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:09.498788 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:09.499136 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:09.499141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:09.499164 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:09.499558 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.499568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.500767 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.500771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.502229 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.502241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.503325 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.503331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.503473 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.503496 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:14:09.503573 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 02:14:09.503700 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":21,"thread_start_us":684,"threads_started":1}
I20260926 02:14:09.504572 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.497852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:40093 (local address 127.28.152.129:40489)
0926 02:14:09.498077 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.498096 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.498846 (+ 750us) server_negotiation.cc:407] Connection header received
0926 02:14:09.499095 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.499101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.499183 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.499287 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:09.500277 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.501021 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.502741 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.503673 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.503829 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.503898 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.504285 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.504411 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:14:09.505249 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.497679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:40489 (local address 127.28.152.130:40093)
0926 02:14:09.498721 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:09.498739 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:09.498873 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:09.499308 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:09.499313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:09.499338 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:09.500105 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.500118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.501166 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.501170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.502582 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.502594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.503931 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.503938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.504110 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.504143 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:14:09.504737 (+ 594us) client_negotiation.cc:240] Negotiation successful
0926 02:14:09.504880 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":54,"thread_start_us":536,"threads_started":1}
I20260926 02:14:09.505203 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380a03b192004d9ca4f0d31d676f6206" candidate_uuid: "c49afa0c59cd4109bd029349db882ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "320e0037c6754a51b79146fc5f118f45" is_pre_election: true
I20260926 02:14:09.505383 4263 raft_consensus.cc:482] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:14:09.505676 4263 raft_consensus.cc:504] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } } peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } }
I20260926 02:14:09.506225 3889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380a03b192004d9ca4f0d31d676f6206" candidate_uuid: "c49afa0c59cd4109bd029349db882ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e34eff83373477c97735ff40f839576" is_pre_election: true
I20260926 02:14:09.506395 4176 raft_consensus.cc:2454] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c49afa0c59cd4109bd029349db882ecd in term 0.
I20260926 02:14:09.506743 3889 raft_consensus.cc:2454] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c49afa0c59cd4109bd029349db882ecd in term 0.
I20260926 02:14:09.507038 3968 leader_election.cc:302] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 320e0037c6754a51b79146fc5f118f45, c49afa0c59cd4109bd029349db882ecd; no voters:
I20260926 02:14:09.507421 4261 raft_consensus.cc:2781] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:14:09.507562 4261 raft_consensus.cc:482] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:14:09.507654 4261 raft_consensus.cc:3037] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:09.507908 4263 leader_election.cc:288] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c49afa0c59cd4109bd029349db882ecd (127.28.152.130:34051), 8e34eff83373477c97735ff40f839576 (127.28.152.129:40489)
I20260926 02:14:09.510741 4261 raft_consensus.cc:504] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } } peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } }
I20260926 02:14:09.511444 4261 leader_election.cc:288] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [CANDIDATE]: Term 1 election: Requested vote from peers 8e34eff83373477c97735ff40f839576 (127.28.152.129:40489), 320e0037c6754a51b79146fc5f118f45 (127.28.152.131:36667)
I20260926 02:14:09.512025 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380a03b192004d9ca4f0d31d676f6206" candidate_uuid: "c49afa0c59cd4109bd029349db882ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "320e0037c6754a51b79146fc5f118f45"
I20260926 02:14:09.512207 4176 raft_consensus.cc:3037] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:09.513789 3889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380a03b192004d9ca4f0d31d676f6206" candidate_uuid: "c49afa0c59cd4109bd029349db882ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e34eff83373477c97735ff40f839576"
I20260926 02:14:09.514029 3889 raft_consensus.cc:3037] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:09.515465 4176 raft_consensus.cc:2454] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c49afa0c59cd4109bd029349db882ecd in term 1.
I20260926 02:14:09.515950 3968 leader_election.cc:302] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 320e0037c6754a51b79146fc5f118f45, c49afa0c59cd4109bd029349db882ecd; no voters:
I20260926 02:14:09.516048 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.507955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:40727 (local address 127.28.152.130:34051)
0926 02:14:09.508135 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.508140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.508269 (+ 129us) server_negotiation.cc:407] Connection header received
0926 02:14:09.508407 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.508416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.508486 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.508578 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:09.508928 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.509977 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.513212 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.514466 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.514625 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.514698 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.515539 (+ 841us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.515683 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:14:09.516446 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.507719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:34051 (local address 127.28.152.131:40727)
0926 02:14:09.508196 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:09.508210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:09.508304 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:09.508597 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:09.508602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:09.508624 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:09.508790 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.508801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.510147 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.510154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.512979 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.513001 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.514663 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.514670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.514996 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.515050 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 02:14:09.515156 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:14:09.515316 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":417}
I20260926 02:14:09.516748 4261 raft_consensus.cc:2781] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:09.517340 4261 raft_consensus.cc:686] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 1 LEADER]: Becoming Leader. State: Replica: c49afa0c59cd4109bd029349db882ecd, State: Running, Role: LEADER
I20260926 02:14:09.517592 4035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380a03b192004d9ca4f0d31d676f6206" candidate_uuid: "320e0037c6754a51b79146fc5f118f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c49afa0c59cd4109bd029349db882ecd" is_pre_election: true
I20260926 02:14:09.518006 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.507719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:40489 (local address 127.28.152.131:47729)
0926 02:14:09.507845 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:09.507862 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:09.507981 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:09.508769 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:09.508773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:09.508793 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:09.508933 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.508943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.509987 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.509992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.512979 (+ 2987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:09.513001 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.516458 (+ 3457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.516465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:09.516697 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.516733 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:14:09.516833 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:14:09.516980 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":243}
I20260926 02:14:09.517961 4261 consensus_queue.cc:237] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } } peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } }
I20260926 02:14:09.518867 4252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.508323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:47729 (local address 127.28.152.129:40489)
0926 02:14:09.508463 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.508467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.508479 (+ 12us) server_negotiation.cc:407] Connection header received
0926 02:14:09.508519 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.508522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.508581 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.508653 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:09.509050 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.509860 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.514790 (+ 4930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.516071 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.516250 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.516347 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.518358 (+ 2011us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.518510 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:14:09.519450 3888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380a03b192004d9ca4f0d31d676f6206" candidate_uuid: "320e0037c6754a51b79146fc5f118f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e34eff83373477c97735ff40f839576" is_pre_election: true
I20260926 02:14:09.520219 3889 raft_consensus.cc:2454] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c49afa0c59cd4109bd029349db882ecd in term 1.
I20260926 02:14:09.521068 4111 leader_election.cc:302] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 320e0037c6754a51b79146fc5f118f45; no voters: 8e34eff83373477c97735ff40f839576, c49afa0c59cd4109bd029349db882ecd
I20260926 02:14:09.521814 4263 raft_consensus.cc:2726] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:14:09.523518 2969 catalog_manager.cc:5495] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to c49afa0c59cd4109bd029349db882ecd (127.28.152.130). New cstate: current_term: 1 leader_uuid: "c49afa0c59cd4109bd029349db882ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } health_report { overall_health: UNKNOWN } } }
I20260926 02:14:09.611809 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.606581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.28.152.130:34051)
0926 02:14:09.606773 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.606779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.606798 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:14:09.606964 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.606968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.607060 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.607188 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:14:09.607527 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.608268 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.609714 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.610122 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.610229 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.610691 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:14:09.611210 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:14:09.611360 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.611450 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.611534 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:14:09.620074 3889 raft_consensus.cc:1261] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576 [term 1 FOLLOWER]: Refusing update from remote peer c49afa0c59cd4109bd029349db882ecd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:09.620124 4176 raft_consensus.cc:1261] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 1 FOLLOWER]: Refusing update from remote peer c49afa0c59cd4109bd029349db882ecd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:09.620968 4266 consensus_queue.cc:1035] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:09.621379 4261 consensus_queue.cc:1035] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:09.653813 2968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:09.654048 2968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:09.660192 2968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 248a9ebcadfd4363957532d4e5a993ec: Sending DeleteTablet for 3 replicas of tablet 380a03b192004d9ca4f0d31d676f6206
I20260926 02:14:09.661217 4015 tablet_service.cc:1514] Processing DeleteTablet for tablet 380a03b192004d9ca4f0d31d676f6206 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:09 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:52454
I20260926 02:14:09.661278 3869 tablet_service.cc:1514] Processing DeleteTablet for tablet 380a03b192004d9ca4f0d31d676f6206 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:09 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:32936
I20260926 02:14:09.662079 4278 tablet_replica.cc:330] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: stopping tablet replica
I20260926 02:14:09.662257 4156 tablet_service.cc:1514] Processing DeleteTablet for tablet 380a03b192004d9ca4f0d31d676f6206 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:09 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:45576
I20260926 02:14:09.662485 4279 tablet_replica.cc:330] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: stopping tablet replica
I20260926 02:14:09.663072 4278 raft_consensus.cc:2229] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:14:09.663172 4279 raft_consensus.cc:2229] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:09.663739 4278 raft_consensus.cc:2258] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:09.663878 4279 raft_consensus.cc:2258] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:09.666145 4279 ts_tablet_manager.cc:1905] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:09.666409 4278 ts_tablet_manager.cc:1905] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:09.673476 4279 ts_tablet_manager.cc:1918] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:09.673715 4279 log.cc:1196] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/380a03b192004d9ca4f0d31d676f6206
I20260926 02:14:09.674321 4279 ts_tablet_manager.cc:1939] T 380a03b192004d9ca4f0d31d676f6206 P 8e34eff83373477c97735ff40f839576: Deleting consensus metadata
I20260926 02:14:09.676280 2955 catalog_manager.cc:4841] TS 8e34eff83373477c97735ff40f839576 (127.28.152.129:40489): tablet 380a03b192004d9ca4f0d31d676f6206 (table test-table [id=e7437e1780a24db1a3b631db1daa4c47]) successfully deleted
I20260926 02:14:09.677183 4280 tablet_replica.cc:330] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: stopping tablet replica
I20260926 02:14:09.677568 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 2937
I20260926 02:14:09.677587 4280 raft_consensus.cc:2229] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:09.678026 4280 raft_consensus.cc:2258] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:09.678494 4278 ts_tablet_manager.cc:1918] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:09.678696 4278 log.cc:1196] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/380a03b192004d9ca4f0d31d676f6206
I20260926 02:14:09.679262 2999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:09.679378 4278 ts_tablet_manager.cc:1939] T 380a03b192004d9ca4f0d31d676f6206 P c49afa0c59cd4109bd029349db882ecd: Deleting consensus metadata
I20260926 02:14:09.680354 4280 ts_tablet_manager.cc:1905] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:09.687752 4280 ts_tablet_manager.cc:1918] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:09.687923 4280 log.cc:1196] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/380a03b192004d9ca4f0d31d676f6206
I20260926 02:14:09.688465 4280 ts_tablet_manager.cc:1939] T 380a03b192004d9ca4f0d31d676f6206 P 320e0037c6754a51b79146fc5f118f45: Deleting consensus metadata
I20260926 02:14:09.696620 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.664990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.28.152.189:37903)
0926 02:14:09.673603 (+ 8613us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.673611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.676591 (+ 2980us) server_negotiation.cc:407] Connection header received
0926 02:14:09.676665 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.676671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.676739 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.676846 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:09.680584 (+ 3738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.681793 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.683796 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.684520 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.684632 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.688462 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:09.688469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:09.688474 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:09.688517 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:09.692651 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.693692 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.693698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.693702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.693853 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.694296 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.694301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.694305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.694658 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:09.694913 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.696014 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.696246 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8453,"thread_start_us":8276,"threads_started":1}
I20260926 02:14:09.711421 4283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.676169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.28.152.188:45997)
0926 02:14:09.676608 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.676613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.698311 (+ 21698us) server_negotiation.cc:407] Connection header received
0926 02:14:09.698376 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.698381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.698467 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.698552 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:09.700393 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.701208 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.702442 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.702894 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.703007 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.705677 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:09.705688 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:09.705691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:09.705729 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:09.708449 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.709320 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.709326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.709329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.709486 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.709880 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.709886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.709889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.710171 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:09.710406 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.710722 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.710941 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":320,"threads_started":1}
I20260926 02:14:09.816359 2953 catalog_manager.cc:4841] TS 320e0037c6754a51b79146fc5f118f45 (127.28.152.131:36667): tablet 380a03b192004d9ca4f0d31d676f6206 (table test-table [id=e7437e1780a24db1a3b631db1daa4c47]) successfully deleted
I20260926 02:14:09.816686 2953 catalog_manager.cc:4841] TS c49afa0c59cd4109bd029349db882ecd (127.28.152.130:34051): tablet 380a03b192004d9ca4f0d31d676f6206 (table test-table [id=e7437e1780a24db1a3b631db1daa4c47]) successfully deleted
I20260926 02:14:09.816874 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 2937
I20260926 02:14:09.870486 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.843013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.28.152.189:37903)
0926 02:14:09.843363 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.843369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.852454 (+ 9085us) server_negotiation.cc:407] Connection header received
0926 02:14:09.853270 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.853276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.853364 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.853444 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:09.856027 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.857075 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.859488 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.860159 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.860264 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.864207 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:09.864215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:09.864221 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:09.864271 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:09.867380 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.868134 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.868138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.868141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.868285 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.868793 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.868798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.868801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.869243 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:09.869458 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.869990 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.870187 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260926 02:14:09.871848 4283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:09.843465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.28.152.188:45997)
0926 02:14:09.843838 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:09.843845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:09.853767 (+ 9922us) server_negotiation.cc:407] Connection header received
0926 02:14:09.854037 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:09.854042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:09.854122 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:09.854190 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:09.856631 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.857648 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.859500 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:09.860159 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:09.860267 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:09.865056 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:09.865063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:09.865068 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:09.865110 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:09.868360 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.869050 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.869056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.869060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.869229 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:09.869595 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:09.869600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:09.869603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:09.869933 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:09.870127 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:09.871180 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0926 02:14:09.871429 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260926 02:14:10.638849 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:10.637981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.131:50623)
0926 02:14:10.638505 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:10.638642 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":260,"threads_started":1}
W20260926 02:14:10.639328 4221 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:34891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
I20260926 02:14:10.644040 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:10.643375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.130:52645)
0926 02:14:10.643763 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:10.643881 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":197,"threads_started":1}
W20260926 02:14:10.644402 4080 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:34891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
I20260926 02:14:10.649785 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:10.649065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.129:45483)
0926 02:14:10.649523 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:10.649631 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
W20260926 02:14:10.650202 3934 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:34891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
I20260926 02:14:11.160883 4295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 248a9ebcadfd4363957532d4e5a993ec)
I20260926 02:14:11.160960 4296 raft_consensus.cc:482] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 248a9ebcadfd4363957532d4e5a993ec)
I20260926 02:14:11.161170 4295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } }
I20260926 02:14:11.161296 4296 raft_consensus.cc:504] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } }
I20260926 02:14:11.162544 4295 leader_election.cc:288] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 248a9ebcadfd4363957532d4e5a993ec (127.28.152.190:34891), f31a651ed7d94d5caa654cd28802f45e (127.28.152.189:37903)
I20260926 02:14:11.168715 4296 leader_election.cc:288] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 248a9ebcadfd4363957532d4e5a993ec (127.28.152.190:34891), 40341a9abb464c30ae76521f28ee3be3 (127.28.152.188:45997)
I20260926 02:14:11.176111 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.162665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:35992)
0926 02:14:11.175574 (+ 12909us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.175788 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12779,"mutex_wait_us":300,"thread_start_us":11547,"threads_started":1}
I20260926 02:14:11.176280 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.162662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:35994)
0926 02:14:11.175167 (+ 12505us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.176038 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12380,"thread_start_us":12076,"threads_started":1}
W20260926 02:14:11.176410 3553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
W20260926 02:14:11.176750 3742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
I20260926 02:14:11.183118 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.162657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:45997 (local address 127.0.0.1:60928)
0926 02:14:11.175444 (+ 12787us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.175463 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:11.175612 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:11.176605 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:11.176610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:11.176637 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:11.176998 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.177011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.178819 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.178825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.180852 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.180862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.182420 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.182426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.182600 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.182633 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:14:11.182700 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 02:14:11.182821 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12676,"thread_start_us":11842,"threads_started":1}
I20260926 02:14:11.183118 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.163082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:37903 (local address 127.0.0.1:43748)
0926 02:14:11.175211 (+ 12129us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.175232 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:11.175403 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:11.176467 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:11.176471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:11.176494 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:11.176826 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.176840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.178507 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.178513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.180632 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.180648 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.182358 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.182365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.182526 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.182564 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:14:11.182631 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 02:14:11.182767 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12014,"thread_start_us":12421,"threads_started":1}
I20260926 02:14:11.183979 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.175384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.28.152.189:37903)
0926 02:14:11.176049 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:11.176055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:11.176079 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:14:11.176150 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:11.176155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:11.176222 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:11.176318 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:11.177026 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.178351 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.180855 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.182141 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.183421 (+ 1280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.183489 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:11.183538 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 02:14:11.183655 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":371,"threads_started":1}
I20260926 02:14:11.184679 3577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40341a9abb464c30ae76521f28ee3be3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f31a651ed7d94d5caa654cd28802f45e" is_pre_election: true
I20260926 02:14:11.184980 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.184530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:36020)
0926 02:14:11.184741 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.184840 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 02:14:11.185083 3577 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40341a9abb464c30ae76521f28ee3be3 in term 1.
W20260926 02:14:11.185230 3742 leader_election.cc:334] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 248a9ebcadfd4363957532d4e5a993ec (127.28.152.190:34891): Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
I20260926 02:14:11.185642 3743 leader_election.cc:302] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40341a9abb464c30ae76521f28ee3be3, f31a651ed7d94d5caa654cd28802f45e; no voters: 248a9ebcadfd4363957532d4e5a993ec
I20260926 02:14:11.185777 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.185412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:36034)
0926 02:14:11.185566 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.185647 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 02:14:11.185777 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.175063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.28.152.188:45997)
0926 02:14:11.176076 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:11.176081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:11.176098 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:14:11.176166 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:11.176170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:11.176452 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:11.177271 (+ 819us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:11.177483 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.178673 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.181028 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.182253 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.183864 (+ 1611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.183959 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:11.184007 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 02:14:11.184856 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":1449,"threads_started":1}
I20260926 02:14:11.186048 4295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 02:14:11.186187 4295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 248a9ebcadfd4363957532d4e5a993ec)
W20260926 02:14:11.186211 3553 leader_election.cc:334] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 248a9ebcadfd4363957532d4e5a993ec (127.28.152.190:34891): Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
I20260926 02:14:11.186287 4295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:14:11.186308 3766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f31a651ed7d94d5caa654cd28802f45e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "40341a9abb464c30ae76521f28ee3be3" is_pre_election: true
I20260926 02:14:11.188304 4295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } }
I20260926 02:14:11.188706 3766 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f31a651ed7d94d5caa654cd28802f45e in current term 2: Already voted for candidate 40341a9abb464c30ae76521f28ee3be3 in this term.
I20260926 02:14:11.188894 4295 leader_election.cc:288] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [CANDIDATE]: Term 2 election: Requested vote from peers 248a9ebcadfd4363957532d4e5a993ec (127.28.152.190:34891), f31a651ed7d94d5caa654cd28802f45e (127.28.152.189:37903)
I20260926 02:14:11.189397 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.188995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:36042)
0926 02:14:11.189114 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.189222 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 02:14:11.189502 3577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40341a9abb464c30ae76521f28ee3be3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f31a651ed7d94d5caa654cd28802f45e"
I20260926 02:14:11.189730 3577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:14:11.190282 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.189973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:36052)
0926 02:14:11.190115 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.190189 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 02:14:11.190474 3742 leader_election.cc:334] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 248a9ebcadfd4363957532d4e5a993ec (127.28.152.190:34891): Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
I20260926 02:14:11.190449 3551 leader_election.cc:302] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f31a651ed7d94d5caa654cd28802f45e; no voters: 248a9ebcadfd4363957532d4e5a993ec, 40341a9abb464c30ae76521f28ee3be3
I20260926 02:14:11.192121 3577 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40341a9abb464c30ae76521f28ee3be3 in term 2.
I20260926 02:14:11.192306 4296 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:14:11.192520 3743 leader_election.cc:302] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 40341a9abb464c30ae76521f28ee3be3, f31a651ed7d94d5caa654cd28802f45e; no voters: 248a9ebcadfd4363957532d4e5a993ec
I20260926 02:14:11.192801 4295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:14:11.193256 4295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [term 2 LEADER]: Becoming Leader. State: Replica: 40341a9abb464c30ae76521f28ee3be3, State: Running, Role: LEADER
I20260926 02:14:11.193609 4295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } }
I20260926 02:14:11.195431 4305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40341a9abb464c30ae76521f28ee3be3. Latest consensus state: current_term: 2 leader_uuid: "40341a9abb464c30ae76521f28ee3be3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:11.195894 4305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:11.196614 4307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:14:11.200054 4307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:14:11.200850 4307 catalog_manager.cc:1259] Loaded cluster ID: aa430a297210411e86ecefbf19f28df6
I20260926 02:14:11.200927 4307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:14:11.202127 4307 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:14:11.202898 4307 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: Loaded TSK: 0
I20260926 02:14:11.203737 4307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:14:11.255301 3756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:14:11.262207 3756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:14:11.266448 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.265971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:36080)
0926 02:14:11.266126 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.266233 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 02:14:11.266544 3577 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [term 2 FOLLOWER]: Refusing update from remote peer 40341a9abb464c30ae76521f28ee3be3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 02:14:11.267366 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.267034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:36090)
0926 02:14:11.267177 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.267248 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 02:14:11.267284 4305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 02:14:11.267545 3742 consensus_peers.cc:487] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 -> Peer 248a9ebcadfd4363957532d4e5a993ec (127.28.152.190:34891): Couldn't send request to peer 248a9ebcadfd4363957532d4e5a993ec. Status: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 02:14:11.270090 4296 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40341a9abb464c30ae76521f28ee3be3. Latest consensus state: current_term: 2 leader_uuid: "40341a9abb464c30ae76521f28ee3be3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:11.270340 4296 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:14:11.272271 4295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "40341a9abb464c30ae76521f28ee3be3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:11.272591 4295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:11.277681 4296 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "40341a9abb464c30ae76521f28ee3be3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "248a9ebcadfd4363957532d4e5a993ec" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34891 } } peers { permanent_uuid: "f31a651ed7d94d5caa654cd28802f45e" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 37903 } attrs { promote: false } } peers { permanent_uuid: "40341a9abb464c30ae76521f28ee3be3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 45997 } attrs { promote: false } } }
I20260926 02:14:11.277966 4296 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31a651ed7d94d5caa654cd28802f45e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:14:11.289738 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.281994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:40489 (local address 127.0.0.1:49534)
0926 02:14:11.282155 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.282174 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:11.282286 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:11.282896 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:11.282902 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:11.282927 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:11.283138 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.283152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.284794 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.284801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.287013 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.287033 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.288830 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.288838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.289067 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.289112 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 02:14:11.289201 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 02:14:11.289362 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 02:14:11.291173 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.282099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.28.152.129:40489)
0926 02:14:11.282547 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:11.282553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:11.282571 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:14:11.282640 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:11.282645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:11.282717 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:11.283275 (+ 558us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:11.283451 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.284624 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.287264 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.288595 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.290559 (+ 1964us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.290646 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:11.290705 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 02:14:11.290876 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":242,"threads_started":1}
I20260926 02:14:11.292025 3869 tablet_service.cc:1467] Processing CreateTablet for tablet d31b62c2951b42e9ab5ddb49c9dbf291 (DEFAULT_TABLE table=test-table [id=d36c1c0dd45b4fdfbee8758c8b4d776f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:11.292551 3869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31b62c2951b42e9ab5ddb49c9dbf291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:11.298992 4318 tablet_bootstrap.cc:492] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576: Bootstrap starting.
I20260926 02:14:11.299832 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.282717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:36667 (local address 127.0.0.1:38286)
0926 02:14:11.283293 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.283309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:11.283424 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:11.293686 (+ 10262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:11.293692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:11.293727 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:11.294061 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.294073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.295453 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.295460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.297609 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.297625 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.299052 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.299059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.299238 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.299275 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:14:11.299392 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 02:14:11.299545 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480}
I20260926 02:14:11.300776 4318 tablet_bootstrap.cc:654] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:11.301005 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.282854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38286 (local address 127.28.152.131:36667)
0926 02:14:11.293305 (+ 10451us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:11.293311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:11.293332 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:14:11.293409 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:11.293415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:11.293487 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:11.293597 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:11.294226 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.295292 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.297778 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.298812 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.300354 (+ 1542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.300430 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:11.300484 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 02:14:11.300618 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10331,"thread_start_us":10222,"threads_started":1}
I20260926 02:14:11.302173 4156 tablet_service.cc:1467] Processing CreateTablet for tablet d31b62c2951b42e9ab5ddb49c9dbf291 (DEFAULT_TABLE table=test-table [id=d36c1c0dd45b4fdfbee8758c8b4d776f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:11.302625 4156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31b62c2951b42e9ab5ddb49c9dbf291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:11.302675 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.283028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:34051 (local address 127.0.0.1:57218)
0926 02:14:11.290065 (+ 7037us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.290089 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:11.290209 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:11.297904 (+ 7695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:11.297909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:11.297938 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:11.298100 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.298111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.299406 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.299412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.300669 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.300682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.301913 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.301943 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.302139 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.302176 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:14:11.302246 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 02:14:11.302395 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6897,"thread_start_us":9750,"threads_started":1}
I20260926 02:14:11.303431 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.283128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.28.152.130:34051)
0926 02:14:11.297607 (+ 14479us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:11.297614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:11.297633 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:14:11.297697 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:11.297703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:11.297770 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:11.297867 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:11.298253 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.299225 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.300835 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.301742 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.302906 (+ 1164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.302977 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:11.303030 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 02:14:11.303166 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14359,"thread_start_us":14290,"threads_started":1}
I20260926 02:14:11.303644 4318 tablet_bootstrap.cc:492] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576: No bootstrap required, opened a new log
I20260926 02:14:11.303793 4318 ts_tablet_manager.cc:1397] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 02:14:11.304422 4318 raft_consensus.cc:348] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } } peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } }
I20260926 02:14:11.304605 4318 raft_consensus.cc:374] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:11.304688 4318 raft_consensus.cc:729] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e34eff83373477c97735ff40f839576, State: Initialized, Role: FOLLOWER
I20260926 02:14:11.304922 4318 consensus_queue.cc:260] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } } peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } }
I20260926 02:14:11.305941 4015 tablet_service.cc:1467] Processing CreateTablet for tablet d31b62c2951b42e9ab5ddb49c9dbf291 (DEFAULT_TABLE table=test-table [id=d36c1c0dd45b4fdfbee8758c8b4d776f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:11.306408 4015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d31b62c2951b42e9ab5ddb49c9dbf291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:11.308858 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.306009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.129:40179)
0926 02:14:11.308378 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.308545 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2219,"threads_started":1}
I20260926 02:14:11.309245 4320 raft_consensus.cc:482] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:14:11.309486 4318 ts_tablet_manager.cc:1428] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 02:14:11.309458 4320 raft_consensus.cc:504] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } } peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } }
I20260926 02:14:11.311619 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.311128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.129:50997)
0926 02:14:11.311364 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.311450 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 02:14:11.311965 3935 heartbeater.cc:503] Master 127.28.152.188:45997 was elected leader, sending a full tablet report...
I20260926 02:14:11.315949 4320 leader_election.cc:288] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 320e0037c6754a51b79146fc5f118f45 (127.28.152.131:36667), c49afa0c59cd4109bd029349db882ecd (127.28.152.130:34051)
I20260926 02:14:11.318948 4322 tablet_bootstrap.cc:492] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45: Bootstrap starting.
I20260926 02:14:11.321394 4322 tablet_bootstrap.cc:654] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:11.323379 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.313196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:49713 (local address 127.28.152.131:36667)
0926 02:14:11.313471 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:11.313477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:11.313498 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:14:11.313576 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:11.313582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:11.313660 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:11.313749 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:11.314492 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.315722 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.320470 (+ 4748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.321926 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.322080 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.322173 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:11.322900 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 02:14:11.323065 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 02:14:11.324091 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.312618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:36667 (local address 127.28.152.129:49713)
0926 02:14:11.312904 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.312926 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:11.313060 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:11.313975 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:11.313980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:11.314035 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:11.314302 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.314313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.315994 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.316001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.320227 (+ 4226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.320250 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.322460 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.322467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.322682 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.322720 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:14:11.323576 (+ 856us) client_negotiation.cc:240] Negotiation successful
0926 02:14:11.323763 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"threads_started":1}
I20260926 02:14:11.325683 4323 tablet_bootstrap.cc:492] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd: Bootstrap starting.
I20260926 02:14:11.327203 4323 tablet_bootstrap.cc:654] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:11.327977 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31b62c2951b42e9ab5ddb49c9dbf291" candidate_uuid: "8e34eff83373477c97735ff40f839576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "320e0037c6754a51b79146fc5f118f45" is_pre_election: true
W20260926 02:14:11.329098 3822 leader_election.cc:341] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 320e0037c6754a51b79146fc5f118f45 (127.28.152.131:36667): Illegal state: must be running to vote when last-logged opid is not known
I20260926 02:14:11.330039 4322 tablet_bootstrap.cc:492] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45: No bootstrap required, opened a new log
I20260926 02:14:11.330189 4322 ts_tablet_manager.cc:1397] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260926 02:14:11.330963 4322 raft_consensus.cc:348] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } } peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } }
I20260926 02:14:11.331197 4322 raft_consensus.cc:374] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:11.331285 4322 raft_consensus.cc:729] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320e0037c6754a51b79146fc5f118f45, State: Initialized, Role: FOLLOWER
I20260926 02:14:11.331547 4322 consensus_queue.cc:260] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } } peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } }
I20260926 02:14:11.334291 4322 ts_tablet_manager.cc:1428] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260926 02:14:11.334532 4222 heartbeater.cc:503] Master 127.28.152.188:45997 was elected leader, sending a full tablet report...
I20260926 02:14:11.338474 4323 tablet_bootstrap.cc:492] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd: No bootstrap required, opened a new log
I20260926 02:14:11.338536 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.332742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.131:43365)
0926 02:14:11.338138 (+ 5396us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.338287 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5266,"thread_start_us":5142,"threads_started":1}
I20260926 02:14:11.338634 4323 ts_tablet_manager.cc:1397] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260926 02:14:11.339399 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.327200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:38021 (local address 127.28.152.130:34051)
0926 02:14:11.327402 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:11.327413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:11.328053 (+ 640us) server_negotiation.cc:407] Connection header received
0926 02:14:11.328489 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:11.328495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:11.328570 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:11.328631 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:11.329601 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.330564 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.336592 (+ 6028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.337906 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.338045 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.338120 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:11.339008 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 02:14:11.339153 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:14:11.339481 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.327104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:34051 (local address 127.28.152.129:38021)
0926 02:14:11.327953 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.327972 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:11.328270 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:11.328808 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:11.328812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:11.328838 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:11.329267 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.329285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.331125 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.331130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.335160 (+ 4030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:11.335181 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.338335 (+ 3154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.338356 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:11.338547 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.338583 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:14:11.338966 (+ 383us) client_negotiation.cc:240] Negotiation successful
0926 02:14:11.339122 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709}
I20260926 02:14:11.339542 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.339153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.131:48119)
0926 02:14:11.339348 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.339423 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 02:14:11.339556 4323 raft_consensus.cc:348] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } } peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } }
I20260926 02:14:11.339802 4323 raft_consensus.cc:374] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:11.339908 4323 raft_consensus.cc:729] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c49afa0c59cd4109bd029349db882ecd, State: Initialized, Role: FOLLOWER
I20260926 02:14:11.340142 4035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31b62c2951b42e9ab5ddb49c9dbf291" candidate_uuid: "8e34eff83373477c97735ff40f839576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c49afa0c59cd4109bd029349db882ecd" is_pre_election: true
I20260926 02:14:11.340152 4323 consensus_queue.cc:260] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } } peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } }
I20260926 02:14:11.340440 4035 raft_consensus.cc:2454] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e34eff83373477c97735ff40f839576 in term 0.
I20260926 02:14:11.340957 3822 leader_election.cc:302] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e34eff83373477c97735ff40f839576, c49afa0c59cd4109bd029349db882ecd; no voters: 320e0037c6754a51b79146fc5f118f45
I20260926 02:14:11.341235 4323 ts_tablet_manager.cc:1428] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 02:14:11.341714 4320 raft_consensus.cc:2781] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:14:11.341840 4320 raft_consensus.cc:482] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:14:11.341935 4320 raft_consensus.cc:3037] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:11.342801 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.341426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.130:57971)
0926 02:14:11.342434 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.342562 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":922,"threads_started":1}
I20260926 02:14:11.344686 4320 raft_consensus.cc:504] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } } peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } }
I20260926 02:14:11.345371 4320 leader_election.cc:288] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [CANDIDATE]: Term 1 election: Requested vote from peers 320e0037c6754a51b79146fc5f118f45 (127.28.152.131:36667), c49afa0c59cd4109bd029349db882ecd (127.28.152.130:34051)
I20260926 02:14:11.345682 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31b62c2951b42e9ab5ddb49c9dbf291" candidate_uuid: "8e34eff83373477c97735ff40f839576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "320e0037c6754a51b79146fc5f118f45"
I20260926 02:14:11.345893 4176 raft_consensus.cc:3037] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:11.346444 4081 heartbeater.cc:503] Master 127.28.152.188:45997 was elected leader, sending a full tablet report...
I20260926 02:14:11.347503 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.346815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.130:48693)
0926 02:14:11.347263 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.347365 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
I20260926 02:14:11.348091 4176 raft_consensus.cc:2454] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e34eff83373477c97735ff40f839576 in term 1.
I20260926 02:14:11.348424 4035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d31b62c2951b42e9ab5ddb49c9dbf291" candidate_uuid: "8e34eff83373477c97735ff40f839576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c49afa0c59cd4109bd029349db882ecd"
I20260926 02:14:11.348490 3822 leader_election.cc:302] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 320e0037c6754a51b79146fc5f118f45, 8e34eff83373477c97735ff40f839576; no voters:
I20260926 02:14:11.348666 4035 raft_consensus.cc:3037] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:11.348790 4320 raft_consensus.cc:2781] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:11.349300 4320 raft_consensus.cc:686] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 1 LEADER]: Becoming Leader. State: Replica: 8e34eff83373477c97735ff40f839576, State: Running, Role: LEADER
I20260926 02:14:11.349709 4320 consensus_queue.cc:237] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } } peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } }
I20260926 02:14:11.350227 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.349828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.129:45681)
0926 02:14:11.349993 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.350081 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 02:14:11.351402 4035 raft_consensus.cc:2454] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e34eff83373477c97735ff40f839576 in term 1.
I20260926 02:14:11.353026 3756 catalog_manager.cc:5495] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e34eff83373477c97735ff40f839576 (127.28.152.129). New cstate: current_term: 1 leader_uuid: "8e34eff83373477c97735ff40f839576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e34eff83373477c97735ff40f839576" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 40489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 } health_report { overall_health: UNKNOWN } } }
I20260926 02:14:11.370837 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.365945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.28.152.129:40489)
0926 02:14:11.366112 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:11.366116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:11.366132 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:14:11.366308 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:11.366312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:11.366394 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:11.366516 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:14:11.366802 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.367565 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.368873 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.369320 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.369417 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.369826 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:14:11.370387 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:14:11.370506 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:11.370549 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 02:14:11.370614 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 02:14:11.375140 4176 raft_consensus.cc:1261] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45 [term 1 FOLLOWER]: Refusing update from remote peer 8e34eff83373477c97735ff40f839576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:11.375415 4035 raft_consensus.cc:1261] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd [term 1 FOLLOWER]: Refusing update from remote peer 8e34eff83373477c97735ff40f839576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:11.375895 4331 consensus_queue.cc:1035] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "320e0037c6754a51b79146fc5f118f45" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:11.376307 4320 consensus_queue.cc:1035] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c49afa0c59cd4109bd029349db882ecd" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:11.378750 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.378160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.131:53447)
0926 02:14:11.378455 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.378568 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 02:14:11.380170 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.379597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.130:33727)
0926 02:14:11.379865 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.379958 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260926 02:14:11.384920 4334 mvcc.cc:205] Tried to move back new op lower bound from 7333432735225081856 to 7333432735132610560. Current Snapshot: MvccSnapshot[applied={T|T < 7333432735225081856}]
I20260926 02:14:11.385887 4336 mvcc.cc:205] Tried to move back new op lower bound from 7333432735225081856 to 7333432735132610560. Current Snapshot: MvccSnapshot[applied={T|T < 7333432735225081856}]
I20260926 02:14:11.387356 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.386713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.131:51261)
0926 02:14:11.387052 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.387166 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20260926 02:14:11.387811 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.387242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.129:45485)
0926 02:14:11.387533 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.387633 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260926 02:14:11.391784 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.391283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.28.152.130:36763)
0926 02:14:11.391524 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.391615 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260926 02:14:11.391791 4335 mvcc.cc:205] Tried to move back new op lower bound from 7333432735225081856 to 7333432735132610560. Current Snapshot: MvccSnapshot[applied={T|T < 7333432735225081856}]
I20260926 02:14:11.403738 3756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:11.404013 3756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:11.415503 3756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40341a9abb464c30ae76521f28ee3be3: Sending DeleteTablet for 3 replicas of tablet d31b62c2951b42e9ab5ddb49c9dbf291
I20260926 02:14:11.416456 3869 tablet_service.cc:1514] Processing DeleteTablet for tablet d31b62c2951b42e9ab5ddb49c9dbf291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:11 UTC) from {username='oryx', principal='oryx/127.28.152.188@KRBTEST.COM'} at 127.0.0.1:49534
I20260926 02:14:11.417987 4344 tablet_replica.cc:330] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576: stopping tablet replica
I20260926 02:14:11.418186 4344 raft_consensus.cc:2229] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:14:11.418399 4156 tablet_service.cc:1514] Processing DeleteTablet for tablet d31b62c2951b42e9ab5ddb49c9dbf291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:11 UTC) from {username='oryx', principal='oryx/127.28.152.188@KRBTEST.COM'} at 127.0.0.1:38286
I20260926 02:14:11.418833 4344 raft_consensus.cc:2258] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:11.420080 4015 tablet_service.cc:1514] Processing DeleteTablet for tablet d31b62c2951b42e9ab5ddb49c9dbf291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:11 UTC) from {username='oryx', principal='oryx/127.28.152.188@KRBTEST.COM'} at 127.0.0.1:57218
I20260926 02:14:11.422474 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 3736
I20260926 02:14:11.424892 4344 ts_tablet_manager.cc:1905] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:11.429572 4345 tablet_replica.cc:330] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45: stopping tablet replica
I20260926 02:14:11.429786 4345 raft_consensus.cc:2229] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:11.430156 4345 raft_consensus.cc:2258] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:11.433709 4345 ts_tablet_manager.cc:1905] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:11.435261 4346 tablet_replica.cc:330] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd: stopping tablet replica
I20260926 02:14:11.440241 4345 ts_tablet_manager.cc:1918] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:11.440389 4345 log.cc:1196] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/d31b62c2951b42e9ab5ddb49c9dbf291
I20260926 02:14:11.440646 4346 raft_consensus.cc:2229] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:11.440904 4345 ts_tablet_manager.cc:1939] T d31b62c2951b42e9ab5ddb49c9dbf291 P 320e0037c6754a51b79146fc5f118f45: Deleting consensus metadata
I20260926 02:14:11.441179 4346 raft_consensus.cc:2258] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:11.442448 4346 ts_tablet_manager.cc:1905] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:11.442775 3740 catalog_manager.cc:4841] TS 320e0037c6754a51b79146fc5f118f45 (127.28.152.131:36667): tablet d31b62c2951b42e9ab5ddb49c9dbf291 (table test-table [id=d36c1c0dd45b4fdfbee8758c8b4d776f]) successfully deleted
I20260926 02:14:11.443570 4344 ts_tablet_manager.cc:1918] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:11.443717 4344 log.cc:1196] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/d31b62c2951b42e9ab5ddb49c9dbf291
I20260926 02:14:11.444232 4344 ts_tablet_manager.cc:1939] T d31b62c2951b42e9ab5ddb49c9dbf291 P 8e34eff83373477c97735ff40f839576: Deleting consensus metadata
I20260926 02:14:11.445420 3742 catalog_manager.cc:4841] TS 8e34eff83373477c97735ff40f839576 (127.28.152.129:40489): tablet d31b62c2951b42e9ab5ddb49c9dbf291 (table test-table [id=d36c1c0dd45b4fdfbee8758c8b4d776f]) successfully deleted
I20260926 02:14:11.449792 4346 ts_tablet_manager.cc:1918] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:11.450039 4346 log.cc:1196] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/d31b62c2951b42e9ab5ddb49c9dbf291
I20260926 02:14:11.450519 4346 ts_tablet_manager.cc:1939] T d31b62c2951b42e9ab5ddb49c9dbf291 P c49afa0c59cd4109bd029349db882ecd: Deleting consensus metadata
I20260926 02:14:11.451742 3740 catalog_manager.cc:4841] TS c49afa0c59cd4109bd029349db882ecd (127.28.152.130:34051): tablet d31b62c2951b42e9ab5ddb49c9dbf291 (table test-table [id=d36c1c0dd45b4fdfbee8758c8b4d776f]) successfully deleted
I20260926 02:14:11.454648 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.423609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.28.152.129:40489)
0926 02:14:11.423792 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:11.423799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:11.436955 (+ 13156us) server_negotiation.cc:407] Connection header received
0926 02:14:11.437464 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:11.437470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:11.437549 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:11.437622 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:11.439498 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.440667 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.442808 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.443412 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.443515 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.447159 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:11.447181 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:11.447202 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:11.447247 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:11.451599 (+ 4352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:11.452510 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:11.452519 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:11.452521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:11.452705 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:11.453106 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:11.453110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:11.453112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:11.453507 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:11.453781 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:11.454094 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 02:14:11.454308 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:14:11.455298 3929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:11.556361 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 3736
I20260926 02:14:11.571182 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 3946
I20260926 02:14:11.584265 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.571953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.28.152.130:34051)
0926 02:14:11.572157 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:11.572162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:11.572180 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:14:11.572288 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:11.572292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:11.572355 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:11.572441 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:11.573771 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.574511 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.575767 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.576163 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.576258 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.578669 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:11.578685 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:11.578701 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:11.578740 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:11.581607 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:11.582311 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:11.582319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:11.582321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:11.582480 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:11.582825 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:11.582829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:11.582831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:11.583256 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:11.583486 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:11.583754 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 02:14:11.583929 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 02:14:11.584956 4075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:11.672758 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.672303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:36104)
0926 02:14:11.672501 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.672591 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 02:14:11.673753 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.673440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34891 (local address 127.0.0.1:36112)
0926 02:14:11.673577 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:11.673645 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 02:14:11.700542 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 3946
I20260926 02:14:11.718885 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 4093
I20260926 02:14:11.731983 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:11.719578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.28.152.131:36667)
0926 02:14:11.719784 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:11.719790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:11.719857 (+ 67us) server_negotiation.cc:407] Connection header received
0926 02:14:11.720045 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:11.720049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:11.720119 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:11.720199 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:11.721694 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.722444 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.723651 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:11.724033 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:11.724139 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:11.726790 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:11.726804 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:11.726822 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:11.726876 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:11.729345 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:11.730044 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:11.730053 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:11.730056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:11.730176 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:11.730525 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:11.730529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:11.730531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:11.730912 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:11.731175 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:11.731462 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 02:14:11.731661 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:14:11.732606 4216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:11.826172 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 4093
I20260926 02:14:11.840126 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 3430
I20260926 02:14:11.841188 3597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:11.956017 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 3430
I20260926 02:14:11.973551 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 3529
I20260926 02:14:11.974731 3786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:12.094061 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 3529
2026-09-26T02:14:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9550 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:14:12 dist-test-slave-hllj krb5kdc[4355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:14:12 dist-test-slave-hllj krb5kdc[4355](info): set up 2 sockets
Sep 26 02:14:12 dist-test-slave-hllj krb5kdc[4355](info): commencing operation
krb5kdc: starting...
W20260926 02:14:13.107234 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:14:13 dist-test-slave-hllj krb5kdc[4355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:14:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:14:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.190.keytab.
Entry for principal oryx/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.190.keytab.
I20260926 02:14:13.321303 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:33767
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.190.keytab
--principal=oryx/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:33767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:13.650156 4368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:13.650483 4368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:13.650619 4368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:13.658077 4368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:14:13.658211 4368 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:14:13.658280 4368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:13.658377 4368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:13.658445 4368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:14:13.658529 4368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:14:13.668524 4368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:33767
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.190.keytab
--principal=oryx/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:33767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:13.669858 4368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:13.671178 4368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:13.682801 4374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:13.682858 4373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:13.682837 4376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:13.683553 4368 server_base.cc:1034] running on GCE node
I20260926 02:14:13.684311 4368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:13.685779 4368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:13.687000 4368 hybrid_clock.cc:648] HybridClock initialized: now 1790388853686937 us; error 37 us; skew 500 ppm
I20260926 02:14:13.687325 4368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:13 dist-test-slave-hllj krb5kdc[4355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388853, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:13.690331 4368 init.cc:377] Logged in from keytab as oryx/127.28.152.190@KRBTEST.COM (short username oryx)
I20260926 02:14:13.691706 4368 webserver.cc:460] Webserver started at http://127.28.152.190:43939/ using document root <none> and password file <none>
I20260926 02:14:13.692394 4368 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:13.692492 4368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:13.692791 4368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:13.695741 4368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "21762cb54a984847b4c717886024eafe"
format_stamp: "Formatted at 2026-09-26 02:14:13 on dist-test-slave-hllj"
server_key: "b12bef3778619ab22b68faf45e558de9"
server_key_iv: "cd8b295359aa4fa8bb1d320dbaa93571"
server_key_version: "encryptionkey@0"
I20260926 02:14:13.696513 4368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "21762cb54a984847b4c717886024eafe"
format_stamp: "Formatted at 2026-09-26 02:14:13 on dist-test-slave-hllj"
server_key: "b12bef3778619ab22b68faf45e558de9"
server_key_iv: "cd8b295359aa4fa8bb1d320dbaa93571"
server_key_version: "encryptionkey@0"
I20260926 02:14:13.701561 4368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:14:13.705304 4383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:13.706832 4368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:14:13.707026 4368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "21762cb54a984847b4c717886024eafe"
format_stamp: "Formatted at 2026-09-26 02:14:13 on dist-test-slave-hllj"
server_key: "b12bef3778619ab22b68faf45e558de9"
server_key_iv: "cd8b295359aa4fa8bb1d320dbaa93571"
server_key_version: "encryptionkey@0"
I20260926 02:14:13.707248 4368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:13.735451 4368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:13.739624 4368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:13.739885 4368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:13.759859 4368 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:33767
I20260926 02:14:13.759886 4444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:33767 every 8 connection(s)
I20260926 02:14:13.761234 4368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:14:13.761713 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 4368
I20260926 02:14:13.761927 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:14:13.762655 29282 external_mini_cluster.cc:1428] Setting key 9b01c51d524bb0980142d0de747fa7c3
I20260926 02:14:13.765664 4445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:14:13 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:13.776098 4445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe: Bootstrap starting.
I20260926 02:14:13.779587 4445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:13.780676 4445 log.cc:824] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:13.783632 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:13.764615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.28.152.190:33767)
0926 02:14:13.765337 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:13.765351 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:13.765398 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:14:13.767029 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:13.767051 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:13.767438 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:13.767791 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:13.769240 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:13.770595 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:13.771837 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:13.772506 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:13.772669 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:13.776433 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:13.776441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:13.776457 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:13.776502 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:13.779476 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:13.780230 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:13.780237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:13.780244 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:13.780426 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:13.780815 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:13.780820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:13.780824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:13.781570 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:13.781872 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:13.782232 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 02:14:13.782585 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":302,"threads_started":1}
I20260926 02:14:13.783834 4445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe: No bootstrap required, opened a new log
I20260926 02:14:13.792176 4445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21762cb54a984847b4c717886024eafe" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33767 } }
I20260926 02:14:13.792650 4445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:13.792753 4445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21762cb54a984847b4c717886024eafe, State: Initialized, Role: FOLLOWER
I20260926 02:14:13.793442 4445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21762cb54a984847b4c717886024eafe" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33767 } }
I20260926 02:14:13.793713 4445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:14:13.793867 4445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:14:13.794057 4445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:13.797178 4445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21762cb54a984847b4c717886024eafe" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33767 } }
I20260926 02:14:13.797783 4445 leader_election.cc:302] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21762cb54a984847b4c717886024eafe; no voters:
I20260926 02:14:13.798393 4445 leader_election.cc:288] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:14:13.798518 4450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:13.799130 4450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [term 1 LEADER]: Becoming Leader. State: Replica: 21762cb54a984847b4c717886024eafe, State: Running, Role: LEADER
I20260926 02:14:13.799716 4450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21762cb54a984847b4c717886024eafe" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33767 } }
I20260926 02:14:13.800688 4445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:14:13.806895 4452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21762cb54a984847b4c717886024eafe. Latest consensus state: current_term: 1 leader_uuid: "21762cb54a984847b4c717886024eafe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21762cb54a984847b4c717886024eafe" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33767 } } }
I20260926 02:14:13.806912 4451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21762cb54a984847b4c717886024eafe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21762cb54a984847b4c717886024eafe" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33767 } } }
I20260926 02:14:13.807351 4452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:13.807571 4451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:13.808918 4459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:14:13.814128 4459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:14:13.822803 4459 catalog_manager.cc:1347] Generated new cluster ID: a076b9a301e249389f579c1eb0a3f41e
I20260926 02:14:13.822921 4459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:14:13.839089 4459 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:14:13.840683 4459 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:14:13.848754 4459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe: Generated new TSK 0
I20260926 02:14:13.849684 4459 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab.
Entry for principal oryx/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab.
I20260926 02:14:13.905494 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33767
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab
--principal=oryx/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:14.236107 4471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:14.236408 4471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:14.236517 4471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:14.244014 4471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:14.244166 4471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:14.244289 4471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:14:14.244378 4471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:14:14.254591 4471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab
--principal=oryx/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.152.190:33767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:14.255928 4471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:14.257354 4471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:14.270533 4479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:14.270597 4477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:14.270488 4476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:14.271828 4471 server_base.cc:1034] running on GCE node
I20260926 02:14:14.272569 4471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:14.273635 4471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:14.274887 4471 hybrid_clock.cc:648] HybridClock initialized: now 1790388854274817 us; error 41 us; skew 500 ppm
I20260926 02:14:14.275233 4471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:14 dist-test-slave-hllj krb5kdc[4355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388854, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:14.278628 4471 init.cc:377] Logged in from keytab as oryx/127.28.152.129@KRBTEST.COM (short username oryx)
I20260926 02:14:14.280107 4471 webserver.cc:460] Webserver started at http://127.28.152.129:44427/ using document root <none> and password file <none>
I20260926 02:14:14.280807 4471 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:14.280912 4471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:14.281236 4471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:14.284416 4471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "a110477956164371a2de473051ebb4bb"
format_stamp: "Formatted at 2026-09-26 02:14:14 on dist-test-slave-hllj"
server_key: "66b4e3849276035522ea4863a6246afe"
server_key_iv: "879b9f8333014a79ec96d18da000940f"
server_key_version: "encryptionkey@0"
I20260926 02:14:14.285332 4471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "a110477956164371a2de473051ebb4bb"
format_stamp: "Formatted at 2026-09-26 02:14:14 on dist-test-slave-hllj"
server_key: "66b4e3849276035522ea4863a6246afe"
server_key_iv: "879b9f8333014a79ec96d18da000940f"
server_key_version: "encryptionkey@0"
I20260926 02:14:14.290575 4471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 02:14:14.294116 4486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:14.295498 4471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:14:14.295646 4471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "a110477956164371a2de473051ebb4bb"
format_stamp: "Formatted at 2026-09-26 02:14:14 on dist-test-slave-hllj"
server_key: "66b4e3849276035522ea4863a6246afe"
server_key_iv: "879b9f8333014a79ec96d18da000940f"
server_key_version: "encryptionkey@0"
I20260926 02:14:14.295883 4471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:14.317607 4471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:14.322168 4471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:14.322464 4471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:14.326674 4471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:14.326769 4471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:14.326921 4471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:14.326992 4471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:14:14 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388854, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:14.351922 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:14.327584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:41511 (local address 127.28.152.190:33767)
0926 02:14:14.328065 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:14.328072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:14.329774 (+ 1702us) server_negotiation.cc:407] Connection header received
0926 02:14:14.332443 (+ 2669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:14.332449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:14.332547 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:14.332660 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:14.335091 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.336101 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.338550 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.339118 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.339232 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:14.344377 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:14.344382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:14.344386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:14.344430 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:14.347696 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:14.349014 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:14.349018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:14.349020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:14.349189 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:14.349761 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:14.349767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:14.349769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:14.350102 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:14.350251 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:14.351302 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0926 02:14:14.351533 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":247,"threads_started":1}
I20260926 02:14:14.353466 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:14.328249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33767 (local address 127.28.152.129:41511)
0926 02:14:14.329111 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:14.329203 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:14.331932 (+ 2729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:14.333000 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:14.333013 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:14.333864 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:14.334838 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:14.334864 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.336337 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.336343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:14.338309 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:14.338329 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.339498 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.339508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:14.339760 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:14.340711 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:14.340749 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:14.344077 (+ 3328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:14.348056 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:14.348068 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:14.348089 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:14.348785 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:14.349306 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:14.349312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:14.349318 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:14.349636 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:14.350320 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:14.350326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:14.350868 (+ 542us) client_negotiation.cc:769] Sending connection context
0926 02:14:14.351191 (+ 323us) client_negotiation.cc:240] Negotiation successful
0926 02:14:14.351838 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":375,"threads_started":1}
I20260926 02:14:14.369306 4471 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:39577
I20260926 02:14:14.369323 4606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:39577 every 8 connection(s)
I20260926 02:14:14.370697 4471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:14:14.373042 4607 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33767
I20260926 02:14:14.373311 4607 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:14.374076 4607 heartbeater.cc:511] Master 127.28.152.190:33767 requested a full tablet report, sending...
I20260926 02:14:14.376372 4400 ts_manager.cc:194] Registered new tserver with Master: a110477956164371a2de473051ebb4bb (127.28.152.129:39577)
I20260926 02:14:14.376732 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 4471
I20260926 02:14:14.376969 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:14:14.377786 29282 external_mini_cluster.cc:1428] Setting key 4c9ec9aeb85c297f08c062498c0e40d4
I20260926 02:14:14.377990 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:14.364415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33767 (local address 127.28.152.129:47647)
0926 02:14:14.365115 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:14.365175 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:14.365365 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:14.365893 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:14.365901 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:14.366493 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:14.366980 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:14.366993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.368118 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.368124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:14.369485 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:14.369501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.370256 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.370262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:14.370444 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:14.371084 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:14.371102 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:14.372072 (+ 970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:14.375148 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:14.375153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:14.375159 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:14.375663 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:14.376023 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:14.376028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:14.376031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:14.376178 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:14.376812 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:14.376817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:14.377012 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 02:14:14.377226 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 02:14:14.377548 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":320,"threads_started":1}
I20260926 02:14:14.378412 4400 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:41511
I20260926 02:14:14.378835 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:14.364506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:47647 (local address 127.28.152.190:33767)
0926 02:14:14.364736 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:14.364744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:14.365287 (+ 543us) server_negotiation.cc:407] Connection header received
0926 02:14:14.365529 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:14.365535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:14.365625 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:14.365730 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:14.367157 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.367970 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.369629 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.370061 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.370176 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:14.372311 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:14.372316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:14.372320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:14.372360 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:14.375023 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:14.375770 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:14.375775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:14.375777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:14.375915 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:14.376288 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:14.376292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:14.376295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:14.376684 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:14.376852 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:14.378259 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0926 02:14:14.378520 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":73}
WARNING: no policy specified for oryx/127.28.152.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab.
Entry for principal oryx/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab.
I20260926 02:14:14.424674 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33767
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab
--principal=oryx/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:14.752434 4611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:14.752771 4611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:14.752912 4611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:14.760692 4611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:14.760840 4611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:14.760967 4611 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:14:14.761075 4611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:14:14.771287 4611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab
--principal=oryx/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.152.190:33767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:14.772792 4611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:14.774493 4611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:14.788645 4617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:14.788688 4616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:14.788626 4619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:14.789530 4611 server_base.cc:1034] running on GCE node
I20260926 02:14:14.790153 4611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:14.791128 4611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:14.792375 4611 hybrid_clock.cc:648] HybridClock initialized: now 1790388854792293 us; error 48 us; skew 500 ppm
I20260926 02:14:14.792675 4611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:14 dist-test-slave-hllj krb5kdc[4355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388854, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:14.796061 4611 init.cc:377] Logged in from keytab as oryx/127.28.152.130@KRBTEST.COM (short username oryx)
I20260926 02:14:14.797662 4611 webserver.cc:460] Webserver started at http://127.28.152.130:40135/ using document root <none> and password file <none>
I20260926 02:14:14.798408 4611 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:14.798502 4611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:14.798851 4611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:14.802099 4611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "9b4f57b618064644b35f926b1d584e5d"
format_stamp: "Formatted at 2026-09-26 02:14:14 on dist-test-slave-hllj"
server_key: "c372de3672fab26b67c65cdd92e1b84c"
server_key_iv: "11c3401cb82f420ce324e15d98414027"
server_key_version: "encryptionkey@0"
I20260926 02:14:14.803153 4611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "9b4f57b618064644b35f926b1d584e5d"
format_stamp: "Formatted at 2026-09-26 02:14:14 on dist-test-slave-hllj"
server_key: "c372de3672fab26b67c65cdd92e1b84c"
server_key_iv: "11c3401cb82f420ce324e15d98414027"
server_key_version: "encryptionkey@0"
I20260926 02:14:14.809180 4611 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:14:14.812914 4626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:14.814468 4611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:14:14.814625 4611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "9b4f57b618064644b35f926b1d584e5d"
format_stamp: "Formatted at 2026-09-26 02:14:14 on dist-test-slave-hllj"
server_key: "c372de3672fab26b67c65cdd92e1b84c"
server_key_iv: "11c3401cb82f420ce324e15d98414027"
server_key_version: "encryptionkey@0"
I20260926 02:14:14.814832 4611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:14.827850 4611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:14.831820 4611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:14.832082 4611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:14.834820 4611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:14.834913 4611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:14.835039 4611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:14.835103 4611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:14:14 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388854, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:14.859345 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:14.835574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:38053 (local address 127.28.152.190:33767)
0926 02:14:14.835750 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:14.835756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:14.837320 (+ 1564us) server_negotiation.cc:407] Connection header received
0926 02:14:14.839561 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:14.839568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:14.839694 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:14.839771 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:14.842426 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.843596 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.846038 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.846636 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.846782 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:14.851781 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:14.851788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:14.851794 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:14.851840 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:14.855265 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:14.856613 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:14.856619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:14.856623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:14.856803 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:14.857447 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:14.857453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:14.857457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:14.857767 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:14.857920 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:14.858716 (+ 796us) server_negotiation.cc:299] Negotiation successful
0926 02:14:14.858963 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:14:14.860391 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:14.836199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33767 (local address 127.28.152.130:38053)
0926 02:14:14.837090 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:14.837174 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:14.839164 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:14.840353 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:14.840363 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:14.841261 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:14.842178 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:14.842211 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.843775 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.843781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:14.845850 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:14.845892 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.846915 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.846928 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:14.847274 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:14.848230 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:14.848267 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:14.851488 (+ 3221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:14.855478 (+ 3990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:14.855492 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:14.855514 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:14.856395 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:14.856962 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:14.856971 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:14.856976 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:14.857311 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:14.857910 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:14.857919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:14.858312 (+ 393us) client_negotiation.cc:769] Sending connection context
0926 02:14:14.858571 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 02:14:14.859145 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":361,"threads_started":1}
I20260926 02:14:14.876981 4611 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:33143
I20260926 02:14:14.877003 4745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:33143 every 8 connection(s)
I20260926 02:14:14.878489 4611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:14:14.880954 4746 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33767
I20260926 02:14:14.881220 4746 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:14.882002 4746 heartbeater.cc:511] Master 127.28.152.190:33767 requested a full tablet report, sending...
I20260926 02:14:14.883427 4398 ts_manager.cc:194] Registered new tserver with Master: 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143)
I20260926 02:14:14.884584 4398 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:38053
I20260926 02:14:14.885051 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 4611
I20260926 02:14:14.885281 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:14:14.885592 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:14.872363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:43249 (local address 127.28.152.190:33767)
0926 02:14:14.872535 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:14.872539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:14.873171 (+ 632us) server_negotiation.cc:407] Connection header received
0926 02:14:14.873366 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:14.873372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:14.873438 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:14.873521 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:14.874938 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.875689 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.877122 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.877623 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.877738 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:14.879817 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:14.879824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:14.879828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:14.879870 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:14.882688 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:14.883354 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:14.883360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:14.883363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:14.883556 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:14.883890 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:14.883896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:14.883899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:14.884331 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:14.884485 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:14.884901 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 02:14:14.885214 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:14:14.885650 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:14.872424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33767 (local address 127.28.152.130:43249)
0926 02:14:14.873039 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:14.873058 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:14.873238 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:14.873572 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:14.873576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:14.874297 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:14.874759 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:14.874776 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.875838 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.875844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:14.876973 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:14.876985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:14.877812 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:14.877819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:14.878000 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:14.878652 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:14.878672 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:14.879575 (+ 903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:14.882831 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:14.882837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:14.882840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:14.883237 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:14.883654 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:14.883658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:14.883660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:14.883782 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:14.884464 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:14.884468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:14.884619 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 02:14:14.884805 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 02:14:14.885182 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":341,"threads_started":1}
I20260926 02:14:14.885805 29282 external_mini_cluster.cc:1428] Setting key e958f41c58d098414dec76f7b8cb9266
WARNING: no policy specified for oryx/127.28.152.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab.
Entry for principal oryx/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab.
I20260926 02:14:14.932304 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33767
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab
--principal=oryx/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:15.261736 4750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:15.262043 4750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:15.262161 4750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:15.269513 4750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:15.269645 4750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:15.269758 4750 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:14:15.269865 4750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:14:15.280082 4750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab
--principal=oryx/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.152.190:33767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:15.281546 4750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:15.282876 4750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:15.296494 4755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:15.296469 4756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:15.296747 4758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:15.297477 4750 server_base.cc:1034] running on GCE node
I20260926 02:14:15.298107 4750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:15.299196 4750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:15.300454 4750 hybrid_clock.cc:648] HybridClock initialized: now 1790388855300388 us; error 47 us; skew 500 ppm
I20260926 02:14:15.300763 4750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:15 dist-test-slave-hllj krb5kdc[4355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388855, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:15.304087 4750 init.cc:377] Logged in from keytab as oryx/127.28.152.131@KRBTEST.COM (short username oryx)
I20260926 02:14:15.305640 4750 webserver.cc:460] Webserver started at http://127.28.152.131:45495/ using document root <none> and password file <none>
I20260926 02:14:15.306377 4750 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:15.306483 4750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:15.306792 4750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:15.310037 4750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "fd011b7327ef47af8963cb038572419d"
format_stamp: "Formatted at 2026-09-26 02:14:15 on dist-test-slave-hllj"
server_key: "42f64fc7716701ed2ab9ec1a154a9270"
server_key_iv: "9e7285a4421c665af134196166913951"
server_key_version: "encryptionkey@0"
I20260926 02:14:15.310906 4750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "fd011b7327ef47af8963cb038572419d"
format_stamp: "Formatted at 2026-09-26 02:14:15 on dist-test-slave-hllj"
server_key: "42f64fc7716701ed2ab9ec1a154a9270"
server_key_iv: "9e7285a4421c665af134196166913951"
server_key_version: "encryptionkey@0"
I20260926 02:14:15.316771 4750 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 02:14:15.320766 4765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:15.322520 4750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:14:15.322711 4750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "fd011b7327ef47af8963cb038572419d"
format_stamp: "Formatted at 2026-09-26 02:14:15 on dist-test-slave-hllj"
server_key: "42f64fc7716701ed2ab9ec1a154a9270"
server_key_iv: "9e7285a4421c665af134196166913951"
server_key_version: "encryptionkey@0"
I20260926 02:14:15.322940 4750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:15.351914 4750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:15.356411 4750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:15.356693 4750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:15.359350 4750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:15.359457 4750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:15.359586 4750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:15.359650 4750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:14:15 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388855, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:15.382195 4607 heartbeater.cc:503] Master 127.28.152.190:33767 was elected leader, sending a full tablet report...
I20260926 02:14:15.385162 4774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.360915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33767 (local address 127.28.152.131:58723)
0926 02:14:15.361752 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:15.361794 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:15.363718 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:15.364603 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:15.364612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:15.365336 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:15.366227 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.366253 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.367603 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.367610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.370166 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.370198 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.371307 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.371314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.371618 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.372469 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:15.372503 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:15.375748 (+ 3245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:15.379890 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:15.379902 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:15.379920 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:15.380447 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:15.380872 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:15.380878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:15.380882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:15.381090 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:15.381760 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:15.381771 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:15.382381 (+ 610us) client_negotiation.cc:769] Sending connection context
0926 02:14:15.382742 (+ 361us) client_negotiation.cc:240] Negotiation successful
0926 02:14:15.383408 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":366,"threads_started":1}
I20260926 02:14:15.386003 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.360255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:58723 (local address 127.28.152.190:33767)
0926 02:14:15.360463 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:15.360468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:15.361921 (+ 1453us) server_negotiation.cc:407] Connection header received
0926 02:14:15.364146 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:15.364153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:15.364253 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:15.364373 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:15.366423 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.367404 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.370387 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.371045 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.371191 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.376206 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:15.376214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:15.376218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:15.376280 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:15.379694 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:15.380606 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:15.380610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:15.380613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:15.380753 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:15.381266 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:15.381271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:15.381273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:15.381594 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:15.381763 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:15.385467 (+ 3704us) server_negotiation.cc:299] Negotiation successful
0926 02:14:15.385703 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:14:15.411033 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.396153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:39511 (local address 127.28.152.190:33767)
0926 02:14:15.396349 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:15.396355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:15.397188 (+ 833us) server_negotiation.cc:407] Connection header received
0926 02:14:15.397331 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:15.397337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:15.397431 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:15.397555 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:15.398450 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.399583 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.401637 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.402341 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.402484 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.404353 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:15.404359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:15.404365 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:15.404411 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:15.408047 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:15.408783 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:15.408789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:15.408793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:15.408974 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:15.409461 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:15.409465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:15.409467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:15.409808 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:15.409997 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:15.410417 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 02:14:15.410632 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:14:15.411154 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.396208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33767 (local address 127.28.152.131:39511)
0926 02:14:15.397014 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:15.397035 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:15.397213 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:15.397574 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:15.397577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:15.398041 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:15.398320 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.398330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.399747 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.399753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.401460 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.401482 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.402548 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.402555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.402695 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.403370 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:15.403393 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:15.404139 (+ 746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:15.408190 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:15.408196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:15.408201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:15.408665 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:15.409084 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:15.409090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:15.409093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:15.409296 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:15.409952 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:15.409957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:15.410158 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 02:14:15.410271 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 02:14:15.410648 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":446,"threads_started":1}
I20260926 02:14:15.421530 4750 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:36181
I20260926 02:14:15.421562 4884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:36181 every 8 connection(s)
I20260926 02:14:15.422731 4750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:14:15.424017 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 4750
I20260926 02:14:15.424202 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:14:15.424682 29282 external_mini_cluster.cc:1428] Setting key 68dc65ed5b4d2bc70093c6303f60b85a
I20260926 02:14:15.424722 4885 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33767
I20260926 02:14:15.424921 4885 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:15.425638 4885 heartbeater.cc:511] Master 127.28.152.190:33767 requested a full tablet report, sending...
I20260926 02:14:15.427039 4399 ts_manager.cc:194] Registered new tserver with Master: fd011b7327ef47af8963cb038572419d (127.28.152.131:36181)
I20260926 02:14:15.428123 4399 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:58723
I20260926 02:14:15.440361 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:14:15 dist-test-slave-hllj krb5kdc[4355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:14:15 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:15 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:14:15.488183 4495 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:15.478112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.28.152.190:33767)
0926 02:14:15.478276 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:15.478280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:15.478553 (+ 273us) server_negotiation.cc:407] Connection header received
0926 02:14:15.478703 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:15.478707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:15.478768 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:15.478859 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:15.480128 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.480903 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.482245 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.482687 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.482779 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.487888 (+ 5109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37468: BlockingRecv error: recv got EOF from 127.0.0.1:37468 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 02:14:15 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:15.505930 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.492058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37482 (local address 127.28.152.190:33767)
0926 02:14:15.492248 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:15.492254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:15.492386 (+ 132us) server_negotiation.cc:407] Connection header received
0926 02:14:15.492596 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:15.492600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:15.492674 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:15.492755 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:15.493969 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.494703 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.496136 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.496563 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.496674 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.500536 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:15.500542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:15.500545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:15.500586 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:15.503266 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:15.504093 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:15.504098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:15.504100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:15.504247 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:15.504658 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:15.504661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:15.504663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:15.504953 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:15.505248 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:15.505468 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 02:14:15.505657 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:14:15.510214 4399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:14:15.517367 4399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:14:15.540860 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.529846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33143 (local address 127.0.0.1:55030)
0926 02:14:15.531301 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:15.531348 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:15.531532 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:15.533737 (+ 2205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:15.533744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:15.533773 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:15.534180 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.534191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.535893 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.535898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.538189 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.538204 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.539878 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.539884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.540065 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.540117 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:14:15.540251 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 02:14:15.540476 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"thread_start_us":1085,"threads_started":1}
I20260926 02:14:15.540850 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.530067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.28.152.130:33143)
0926 02:14:15.532592 (+ 2525us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:15.532601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:15.532631 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:14:15.532735 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:15.532743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:15.533489 (+ 746us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:15.533703 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:15.534405 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.535770 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.538378 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.539700 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.539855 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.540058 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:15.540383 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 02:14:15.540555 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2374,"thread_start_us":2251,"threads_started":1}
I20260926 02:14:15.541427 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.530743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.28.152.131:36181)
0926 02:14:15.532170 (+ 1427us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:15.532188 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:15.533057 (+ 869us) server_negotiation.cc:407] Connection header received
0926 02:14:15.533400 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:15.533409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:15.533606 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:15.533775 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:15.534842 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.536387 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.538772 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.539898 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.540094 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.540260 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:15.540925 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 02:14:15.541098 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1255,"thread_start_us":1070,"threads_started":1}
I20260926 02:14:15.542109 4905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.530034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:36181 (local address 127.0.0.1:53980)
0926 02:14:15.532917 (+ 2883us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:15.532936 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:15.533195 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:15.534158 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:15.534162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:15.534186 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:15.534642 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.534653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.536536 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.536541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.538622 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.538632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.540209 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.540215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.540375 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.540404 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 02:14:15.540478 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 02:14:15.540594 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2751,"thread_start_us":2618,"threads_started":1}
I20260926 02:14:15.542979 4908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.532833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.28.152.129:39577)
0926 02:14:15.534445 (+ 1612us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:15.534453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:15.534480 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:14:15.534551 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:15.534557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:15.534722 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:15.534921 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:15.535436 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.537053 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.539510 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.540879 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.541051 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.541231 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:15.542644 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0926 02:14:15.542772 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1466,"thread_start_us":1347,"threads_started":1}
I20260926 02:14:15.545214 4907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.532351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:39577 (local address 127.0.0.1:44982)
0926 02:14:15.533208 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:15.533224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:15.533313 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:15.534905 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:15.534910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:15.534933 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:15.535249 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.535259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.537212 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.537217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.539328 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.539342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.541360 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.541367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.541631 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.541667 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:14:15.541749 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 02:14:15.541873 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":623,"threads_started":1}
I20260926 02:14:15.544821 4676 tablet_service.cc:1467] Processing CreateTablet for tablet 14200c2e72b84c31b8262f06dd7fe963 (DEFAULT_TABLE table=test-table [id=b49d4835626849888d0aca0718364f6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:15.545286 4819 tablet_service.cc:1467] Processing CreateTablet for tablet 14200c2e72b84c31b8262f06dd7fe963 (DEFAULT_TABLE table=test-table [id=b49d4835626849888d0aca0718364f6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:15.546187 4676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14200c2e72b84c31b8262f06dd7fe963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:15.546777 4819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14200c2e72b84c31b8262f06dd7fe963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:15.549005 4537 tablet_service.cc:1467] Processing CreateTablet for tablet 14200c2e72b84c31b8262f06dd7fe963 (DEFAULT_TABLE table=test-table [id=b49d4835626849888d0aca0718364f6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:15.550411 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14200c2e72b84c31b8262f06dd7fe963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:15.562750 4910 tablet_bootstrap.cc:492] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: Bootstrap starting.
I20260926 02:14:15.562692 4909 tablet_bootstrap.cc:492] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: Bootstrap starting.
I20260926 02:14:15.565256 4911 tablet_bootstrap.cc:492] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: Bootstrap starting.
I20260926 02:14:15.565944 4910 tablet_bootstrap.cc:654] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:15.567312 4910 log.cc:824] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:15.568182 4911 tablet_bootstrap.cc:654] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:15.568512 4909 tablet_bootstrap.cc:654] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:15.569523 4911 log.cc:824] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:15.569765 4909 log.cc:824] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:15.571851 4910 tablet_bootstrap.cc:492] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: No bootstrap required, opened a new log
I20260926 02:14:15.572167 4910 ts_tablet_manager.cc:1397] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 02:14:15.572321 4909 tablet_bootstrap.cc:492] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: No bootstrap required, opened a new log
I20260926 02:14:15.572561 4911 tablet_bootstrap.cc:492] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: No bootstrap required, opened a new log
I20260926 02:14:15.572566 4909 ts_tablet_manager.cc:1397] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 02:14:15.572824 4911 ts_tablet_manager.cc:1397] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 02:14:15.579453 4909 raft_consensus.cc:348] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } }
I20260926 02:14:15.579850 4909 raft_consensus.cc:374] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:15.579948 4909 raft_consensus.cc:729] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd011b7327ef47af8963cb038572419d, State: Initialized, Role: FOLLOWER
I20260926 02:14:15.580633 4909 consensus_queue.cc:260] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } }
I20260926 02:14:15.581924 4885 heartbeater.cc:503] Master 127.28.152.190:33767 was elected leader, sending a full tablet report...
I20260926 02:14:15.582667 4909 ts_tablet_manager.cc:1428] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260926 02:14:15.582756 4911 raft_consensus.cc:348] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } }
I20260926 02:14:15.583216 4911 raft_consensus.cc:374] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:15.583328 4911 raft_consensus.cc:729] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a110477956164371a2de473051ebb4bb, State: Initialized, Role: FOLLOWER
I20260926 02:14:15.583380 4910 raft_consensus.cc:348] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } }
I20260926 02:14:15.583824 4910 raft_consensus.cc:374] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:15.583917 4910 raft_consensus.cc:729] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4f57b618064644b35f926b1d584e5d, State: Initialized, Role: FOLLOWER
I20260926 02:14:15.583936 4911 consensus_queue.cc:260] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } }
I20260926 02:14:15.584496 4910 consensus_queue.cc:260] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } }
I20260926 02:14:15.585708 4911 ts_tablet_manager.cc:1428] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260926 02:14:15.587791 4746 heartbeater.cc:503] Master 127.28.152.190:33767 was elected leader, sending a full tablet report...
I20260926 02:14:15.587824 4910 ts_tablet_manager.cc:1428] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260926 02:14:15.623620 4608 tablet.cc:2369] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:14:15.631062 4747 tablet.cc:2369] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:14:15.674578 4886 tablet.cc:2369] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:14:15.698050 4917 raft_consensus.cc:482] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:14:15.698379 4917 raft_consensus.cc:504] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } }
I20260926 02:14:15.700136 4917 leader_election.cc:288] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd011b7327ef47af8963cb038572419d (127.28.152.131:36181), a110477956164371a2de473051ebb4bb (127.28.152.129:39577)
I20260926 02:14:15.706866 4908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.700656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:47907 (local address 127.28.152.129:39577)
0926 02:14:15.700897 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:15.700904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:15.701626 (+ 722us) server_negotiation.cc:407] Connection header received
0926 02:14:15.701796 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:15.701800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:15.701858 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:15.701948 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:15.702675 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.703357 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.705186 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.705937 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.706098 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.706169 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:15.706520 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 02:14:15.706645 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:14:15.706863 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.700542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:39505 (local address 127.28.152.131:36181)
0926 02:14:15.700754 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:15.700761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:15.701506 (+ 745us) server_negotiation.cc:407] Connection header received
0926 02:14:15.701736 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:15.701741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:15.701813 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:15.701891 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:15.702582 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.703300 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.705186 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.705927 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.706083 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.706159 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:15.706495 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 02:14:15.706619 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:14:15.707057 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.700303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:36181 (local address 127.28.152.130:39505)
0926 02:14:15.701395 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:15.701411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:15.701565 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:15.701962 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:15.701967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:15.702001 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:15.702420 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.702429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.703430 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.703436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.705019 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.705031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.706167 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.706174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.706334 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.706372 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:14:15.706445 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 02:14:15.706580 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":78,"thread_start_us":714,"threads_started":1}
I20260926 02:14:15.707165 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.700580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:39577 (local address 127.28.152.130:47907)
0926 02:14:15.701528 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:15.701546 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:15.701657 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:15.701972 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:15.701976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:15.702064 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:15.702528 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.702540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.703507 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.703511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.705015 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:15.705027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.706206 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.706213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:15.706346 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.706390 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 02:14:15.706478 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 02:14:15.706608 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":42,"spinlock_wait_cycles":26880,"thread_start_us":563,"threads_started":1}
I20260926 02:14:15.708066 4840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14200c2e72b84c31b8262f06dd7fe963" candidate_uuid: "9b4f57b618064644b35f926b1d584e5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd011b7327ef47af8963cb038572419d" is_pre_election: true
I20260926 02:14:15.708395 4557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14200c2e72b84c31b8262f06dd7fe963" candidate_uuid: "9b4f57b618064644b35f926b1d584e5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a110477956164371a2de473051ebb4bb" is_pre_election: true
I20260926 02:14:15.708581 4840 raft_consensus.cc:2454] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b4f57b618064644b35f926b1d584e5d in term 0.
I20260926 02:14:15.708868 4557 raft_consensus.cc:2454] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b4f57b618064644b35f926b1d584e5d in term 0.
I20260926 02:14:15.709306 4629 leader_election.cc:302] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b4f57b618064644b35f926b1d584e5d, fd011b7327ef47af8963cb038572419d; no voters:
I20260926 02:14:15.709707 4917 raft_consensus.cc:2781] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:14:15.709834 4917 raft_consensus.cc:482] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:14:15.709928 4917 raft_consensus.cc:3037] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:15.712090 4917 raft_consensus.cc:504] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } }
I20260926 02:14:15.712661 4917 leader_election.cc:288] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [CANDIDATE]: Term 1 election: Requested vote from peers fd011b7327ef47af8963cb038572419d (127.28.152.131:36181), a110477956164371a2de473051ebb4bb (127.28.152.129:39577)
I20260926 02:14:15.713152 4557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14200c2e72b84c31b8262f06dd7fe963" candidate_uuid: "9b4f57b618064644b35f926b1d584e5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a110477956164371a2de473051ebb4bb"
I20260926 02:14:15.713156 4840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14200c2e72b84c31b8262f06dd7fe963" candidate_uuid: "9b4f57b618064644b35f926b1d584e5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd011b7327ef47af8963cb038572419d"
I20260926 02:14:15.713339 4557 raft_consensus.cc:3037] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:15.713340 4840 raft_consensus.cc:3037] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:15.715513 4557 raft_consensus.cc:2454] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b4f57b618064644b35f926b1d584e5d in term 1.
I20260926 02:14:15.715523 4840 raft_consensus.cc:2454] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b4f57b618064644b35f926b1d584e5d in term 1.
I20260926 02:14:15.715970 4630 leader_election.cc:302] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b4f57b618064644b35f926b1d584e5d, a110477956164371a2de473051ebb4bb; no voters:
I20260926 02:14:15.716267 4917 raft_consensus.cc:2781] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:15.716696 4917 raft_consensus.cc:686] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 1 LEADER]: Becoming Leader. State: Replica: 9b4f57b618064644b35f926b1d584e5d, State: Running, Role: LEADER
I20260926 02:14:15.717226 4917 consensus_queue.cc:237] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } }
I20260926 02:14:15.722039 4398 catalog_manager.cc:5495] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b4f57b618064644b35f926b1d584e5d (127.28.152.130). New cstate: current_term: 1 leader_uuid: "9b4f57b618064644b35f926b1d584e5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } health_report { overall_health: UNKNOWN } } }
I20260926 02:14:15.778702 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:15.773455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.28.152.130:33143)
0926 02:14:15.773659 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:15.773663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:15.773734 (+ 71us) server_negotiation.cc:407] Connection header received
0926 02:14:15.773927 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:15.773931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:15.774013 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:15.774139 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:14:15.774419 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.775139 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.776455 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:15.776878 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:15.776989 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:15.777576 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:14:15.778149 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:14:15.778287 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:15.778366 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 02:14:15.778437 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:14:15.787189 4840 raft_consensus.cc:1261] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Refusing update from remote peer 9b4f57b618064644b35f926b1d584e5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:15.787210 4557 raft_consensus.cc:1261] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Refusing update from remote peer 9b4f57b618064644b35f926b1d584e5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:15.788125 4917 consensus_queue.cc:1035] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:15.788416 4920 consensus_queue.cc:1035] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:15.801978 4924 mvcc.cc:205] Tried to move back new op lower bound from 7333432753293684736 to 7333432753021874176. Current Snapshot: MvccSnapshot[applied={T|T < 7333432753293684736}]
I20260926 02:14:15.815259 4399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:15.815527 4399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:15.819371 4399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe: Sending DeleteTablet for 3 replicas of tablet 14200c2e72b84c31b8262f06dd7fe963
I20260926 02:14:15.820720 4537 tablet_service.cc:1514] Processing DeleteTablet for tablet 14200c2e72b84c31b8262f06dd7fe963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:15 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:44982
I20260926 02:14:15.821041 4676 tablet_service.cc:1514] Processing DeleteTablet for tablet 14200c2e72b84c31b8262f06dd7fe963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:15 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:55030
I20260926 02:14:15.821519 4819 tablet_service.cc:1514] Processing DeleteTablet for tablet 14200c2e72b84c31b8262f06dd7fe963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:15 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:53980
I20260926 02:14:15.821681 4932 tablet_replica.cc:330] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: stopping tablet replica
I20260926 02:14:15.822130 4932 raft_consensus.cc:2229] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:15.822678 4932 raft_consensus.cc:2258] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:15.823138 4399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:14:15.823719 4399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:14:15.827185 4933 tablet_replica.cc:330] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: stopping tablet replica
I20260926 02:14:15.828644 4932 ts_tablet_manager.cc:1905] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:15.827687 4933 raft_consensus.cc:2229] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:14:15.829912 4933 raft_consensus.cc:2258] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:15.832094 4934 tablet_replica.cc:330] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: stopping tablet replica
I20260926 02:14:15.832329 4818 tablet_service.cc:1467] Processing CreateTablet for tablet fba7a556f7924e24a8a1a97aead98827 (DEFAULT_TABLE table=test-table [id=0b2aad7d40aa4c7784c200bbccffe6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:15.832616 4934 raft_consensus.cc:2229] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:15.832890 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba7a556f7924e24a8a1a97aead98827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:15.833240 4934 raft_consensus.cc:2258] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:15.835955 4933 ts_tablet_manager.cc:1905] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:15.836596 4932 ts_tablet_manager.cc:1918] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:15.836835 4932 log.cc:1196] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/14200c2e72b84c31b8262f06dd7fe963
I20260926 02:14:15.837567 4932 ts_tablet_manager.cc:1939] T 14200c2e72b84c31b8262f06dd7fe963 P a110477956164371a2de473051ebb4bb: Deleting consensus metadata
I20260926 02:14:15.837548 4537 tablet_service.cc:1467] Processing CreateTablet for tablet fba7a556f7924e24a8a1a97aead98827 (DEFAULT_TABLE table=test-table [id=0b2aad7d40aa4c7784c200bbccffe6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:15.838073 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba7a556f7924e24a8a1a97aead98827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:15.838171 4934 ts_tablet_manager.cc:1905] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:15.839730 4387 catalog_manager.cc:4841] TS a110477956164371a2de473051ebb4bb (127.28.152.129:39577): tablet 14200c2e72b84c31b8262f06dd7fe963 (table test-table [id=b49d4835626849888d0aca0718364f6e]) successfully deleted
I20260926 02:14:15.842868 4933 ts_tablet_manager.cc:1918] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:15.843052 4933 log.cc:1196] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/14200c2e72b84c31b8262f06dd7fe963
I20260926 02:14:15.843464 4676 tablet_service.cc:1467] Processing CreateTablet for tablet fba7a556f7924e24a8a1a97aead98827 (DEFAULT_TABLE table=test-table [id=0b2aad7d40aa4c7784c200bbccffe6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:15.843593 4933 ts_tablet_manager.cc:1939] T 14200c2e72b84c31b8262f06dd7fe963 P 9b4f57b618064644b35f926b1d584e5d: Deleting consensus metadata
I20260926 02:14:15.843854 4676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba7a556f7924e24a8a1a97aead98827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:15.844920 4934 ts_tablet_manager.cc:1918] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:15.845078 4934 log.cc:1196] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/14200c2e72b84c31b8262f06dd7fe963
I20260926 02:14:15.845611 4934 ts_tablet_manager.cc:1939] T 14200c2e72b84c31b8262f06dd7fe963 P fd011b7327ef47af8963cb038572419d: Deleting consensus metadata
I20260926 02:14:15.847414 4386 catalog_manager.cc:4841] TS fd011b7327ef47af8963cb038572419d (127.28.152.131:36181): tablet 14200c2e72b84c31b8262f06dd7fe963 (table test-table [id=b49d4835626849888d0aca0718364f6e]) successfully deleted
I20260926 02:14:15.849083 4910 tablet_bootstrap.cc:492] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d: Bootstrap starting.
I20260926 02:14:15.849783 4911 tablet_bootstrap.cc:492] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb: Bootstrap starting.
I20260926 02:14:15.850903 4910 tablet_bootstrap.cc:654] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:15.851370 4911 tablet_bootstrap.cc:654] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:15.853799 4909 tablet_bootstrap.cc:492] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d: Bootstrap starting.
I20260926 02:14:15.855288 4911 tablet_bootstrap.cc:492] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb: No bootstrap required, opened a new log
I20260926 02:14:15.855382 4909 tablet_bootstrap.cc:654] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:15.855437 4911 ts_tablet_manager.cc:1397] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260926 02:14:15.855837 4910 tablet_bootstrap.cc:492] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d: No bootstrap required, opened a new log
I20260926 02:14:15.855993 4910 ts_tablet_manager.cc:1397] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260926 02:14:15.856149 4911 raft_consensus.cc:348] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:15.856380 4911 raft_consensus.cc:374] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:15.856469 4911 raft_consensus.cc:729] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a110477956164371a2de473051ebb4bb, State: Initialized, Role: FOLLOWER
I20260926 02:14:15.856685 4385 catalog_manager.cc:4841] TS 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143): tablet 14200c2e72b84c31b8262f06dd7fe963 (table test-table [id=b49d4835626849888d0aca0718364f6e]) successfully deleted
I20260926 02:14:15.856752 4911 consensus_queue.cc:260] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:15.856796 4910 raft_consensus.cc:348] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:15.857005 4910 raft_consensus.cc:374] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:15.857096 4910 raft_consensus.cc:729] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4f57b618064644b35f926b1d584e5d, State: Initialized, Role: FOLLOWER
I20260926 02:14:15.857558 4910 consensus_queue.cc:260] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:15.859297 4909 tablet_bootstrap.cc:492] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d: No bootstrap required, opened a new log
I20260926 02:14:15.859455 4909 ts_tablet_manager.cc:1397] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260926 02:14:15.859596 4911 ts_tablet_manager.cc:1428] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260926 02:14:15.860257 4909 raft_consensus.cc:348] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:15.860460 4909 raft_consensus.cc:374] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:15.860549 4909 raft_consensus.cc:729] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd011b7327ef47af8963cb038572419d, State: Initialized, Role: FOLLOWER
I20260926 02:14:15.860791 4909 consensus_queue.cc:260] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:15.861250 4909 ts_tablet_manager.cc:1428] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 02:14:15.861621 4910 ts_tablet_manager.cc:1428] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 02:14:16.045758 4915 raft_consensus.cc:482] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:14:16.045960 4915 raft_consensus.cc:504] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.047469 4915 leader_election.cc:288] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a110477956164371a2de473051ebb4bb (127.28.152.129:39577), 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143)
I20260926 02:14:16.055610 4908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.047914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:49213 (local address 127.28.152.129:39577)
0926 02:14:16.048112 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.048119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.049162 (+ 1043us) server_negotiation.cc:407] Connection header received
0926 02:14:16.049494 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.049500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.049558 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.049629 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:16.050114 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.051235 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.053822 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.054626 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.054791 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.054869 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:16.055139 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 02:14:16.055276 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:14:16.055678 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.047756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:39577 (local address 127.28.152.131:49213)
0926 02:14:16.049005 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:16.049025 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:16.049275 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:16.049664 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:16.049669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:16.049695 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:16.049922 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.049934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.051384 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.051388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.053600 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.053619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.054798 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.054804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.054981 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.055006 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:14:16.055091 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 02:14:16.055239 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"mutex_wait_us":9,"thread_start_us":1249,"threads_started":1}
I20260926 02:14:16.056444 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.048379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:33755 (local address 127.28.152.130:33143)
0926 02:14:16.048571 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.048577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.049381 (+ 804us) server_negotiation.cc:407] Connection header received
0926 02:14:16.049501 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.049507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.049609 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.049713 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:16.050320 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.051119 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.053791 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.055123 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.055283 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.055358 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:16.055978 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 02:14:16.056136 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:14:16.056643 4557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba7a556f7924e24a8a1a97aead98827" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a110477956164371a2de473051ebb4bb" is_pre_election: true
I20260926 02:14:16.056945 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.047824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33143 (local address 127.28.152.131:33755)
0926 02:14:16.049271 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:16.049287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:16.049382 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:16.049801 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:16.049805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:16.049830 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:16.050162 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.050172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.051257 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.051263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.053600 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.053619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.055556 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.055564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.055758 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.055792 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:14:16.056460 (+ 668us) client_negotiation.cc:240] Negotiation successful
0926 02:14:16.056614 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"mutex_wait_us":116,"thread_start_us":906,"threads_started":1}
I20260926 02:14:16.056913 4557 raft_consensus.cc:2454] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 0.
I20260926 02:14:16.057520 4769 leader_election.cc:302] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a110477956164371a2de473051ebb4bb, fd011b7327ef47af8963cb038572419d; no voters:
I20260926 02:14:16.058015 4915 raft_consensus.cc:2781] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:14:16.058127 4915 raft_consensus.cc:482] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:14:16.058212 4915 raft_consensus.cc:3037] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:16.060292 4915 raft_consensus.cc:504] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.060681 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba7a556f7924e24a8a1a97aead98827" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4f57b618064644b35f926b1d584e5d" is_pre_election: true
I20260926 02:14:16.060925 4915 leader_election.cc:288] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 election: Requested vote from peers a110477956164371a2de473051ebb4bb (127.28.152.129:39577), 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143)
I20260926 02:14:16.061282 4696 raft_consensus.cc:2454] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 0.
I20260926 02:14:16.061518 4557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba7a556f7924e24a8a1a97aead98827" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a110477956164371a2de473051ebb4bb"
I20260926 02:14:16.061730 4557 raft_consensus.cc:3037] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:16.061685 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba7a556f7924e24a8a1a97aead98827" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4f57b618064644b35f926b1d584e5d"
I20260926 02:14:16.061959 4695 raft_consensus.cc:3037] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:16.064651 4557 raft_consensus.cc:2454] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 1.
I20260926 02:14:16.065182 4769 leader_election.cc:302] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a110477956164371a2de473051ebb4bb, fd011b7327ef47af8963cb038572419d; no voters:
I20260926 02:14:16.065387 4695 raft_consensus.cc:2454] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 1.
I20260926 02:14:16.065505 4915 raft_consensus.cc:2781] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:16.066071 4915 raft_consensus.cc:686] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 1 LEADER]: Becoming Leader. State: Replica: fd011b7327ef47af8963cb038572419d, State: Running, Role: LEADER
I20260926 02:14:16.066515 4915 consensus_queue.cc:237] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.070448 4399 catalog_manager.cc:5495] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d reported cstate change: term changed from 0 to 1, leader changed from <none> to fd011b7327ef47af8963cb038572419d (127.28.152.131). New cstate: current_term: 1 leader_uuid: "fd011b7327ef47af8963cb038572419d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } health_report { overall_health: HEALTHY } } }
Sep 26 02:14:16 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388853, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.190@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:16.171200 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.155521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33767 (local address 127.0.0.1:37488)
0926 02:14:16.156793 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:16.156813 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:16.157035 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:16.157480 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:16.157485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:16.158573 (+ 1088us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:16.158803 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.158816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.159927 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.159933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.161490 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.161505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.162194 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.162201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.162380 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.163159 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:16.163183 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:16.165789 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:16.168710 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:16.168724 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:16.168733 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:16.169173 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:16.169503 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:16.169509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:16.169512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:16.169643 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:16.170148 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:16.170157 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:16.170316 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 02:14:16.170453 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 02:14:16.170743 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"thread_start_us":991,"threads_started":1}
I20260926 02:14:16.171257 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.155547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.28.152.190:33767)
0926 02:14:16.156492 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.156498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.156930 (+ 432us) server_negotiation.cc:407] Connection header received
0926 02:14:16.157224 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.157228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.157303 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.157396 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:16.158985 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.159757 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.161636 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.162023 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.162112 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.166006 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:16.166011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:16.166014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:16.166043 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:16.168583 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:16.169286 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:16.169291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:16.169294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:16.169402 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:16.169746 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:16.169749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:16.169751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:16.170027 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:16.170160 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:16.170574 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 02:14:16.170882 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":785,"thread_start_us":575,"threads_started":1}
I20260926 02:14:16.176451 4399 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:37488:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:14:16.177619 4399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:14:16.184551 4537 tablet_service.cc:1467] Processing CreateTablet for tablet b5fad187480440c3b95e8863087286a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5c6493cb1ef410385998138a78129b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:14:16.184911 4818 tablet_service.cc:1467] Processing CreateTablet for tablet b5fad187480440c3b95e8863087286a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5c6493cb1ef410385998138a78129b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:14:16.185214 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5fad187480440c3b95e8863087286a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:16.184943 4676 tablet_service.cc:1467] Processing CreateTablet for tablet b5fad187480440c3b95e8863087286a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a5c6493cb1ef410385998138a78129b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:14:16.185432 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5fad187480440c3b95e8863087286a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:16.185623 4676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5fad187480440c3b95e8863087286a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:16.192644 4911 tablet_bootstrap.cc:492] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: Bootstrap starting.
I20260926 02:14:16.192874 4909 tablet_bootstrap.cc:492] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d: Bootstrap starting.
I20260926 02:14:16.194389 4911 tablet_bootstrap.cc:654] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:16.194389 4909 tablet_bootstrap.cc:654] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:16.195483 4910 tablet_bootstrap.cc:492] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d: Bootstrap starting.
I20260926 02:14:16.197503 4910 tablet_bootstrap.cc:654] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:16.209244 4910 tablet_bootstrap.cc:492] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d: No bootstrap required, opened a new log
I20260926 02:14:16.209335 4909 tablet_bootstrap.cc:492] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d: No bootstrap required, opened a new log
I20260926 02:14:16.209465 4910 ts_tablet_manager.cc:1397] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260926 02:14:16.209698 4909 ts_tablet_manager.cc:1397] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260926 02:14:16.210323 4910 raft_consensus.cc:348] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.210605 4910 raft_consensus.cc:374] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:16.210688 4910 raft_consensus.cc:729] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4f57b618064644b35f926b1d584e5d, State: Initialized, Role: FOLLOWER
I20260926 02:14:16.210536 4909 raft_consensus.cc:348] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.211001 4909 raft_consensus.cc:374] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:16.211167 4909 raft_consensus.cc:729] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd011b7327ef47af8963cb038572419d, State: Initialized, Role: FOLLOWER
I20260926 02:14:16.211091 4910 consensus_queue.cc:260] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.211509 4909 consensus_queue.cc:260] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.211695 4910 ts_tablet_manager.cc:1428] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:14:16.211578 4917 tablet_replica.cc:439] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } }
I20260926 02:14:16.211865 4917 tablet_replica.cc:442] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:14:16.212110 4915 tablet_replica.cc:439] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } }
I20260926 02:14:16.212390 4915 tablet_replica.cc:442] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:14:16.213349 4909 ts_tablet_manager.cc:1428] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 02:14:16.215583 4911 tablet_bootstrap.cc:492] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: No bootstrap required, opened a new log
I20260926 02:14:16.215773 4911 ts_tablet_manager.cc:1397] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.007s
I20260926 02:14:16.216535 4911 raft_consensus.cc:348] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.216787 4911 raft_consensus.cc:374] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:16.216863 4911 raft_consensus.cc:729] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a110477956164371a2de473051ebb4bb, State: Initialized, Role: FOLLOWER
I20260926 02:14:16.217101 4911 consensus_queue.cc:260] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.217747 4911 ts_tablet_manager.cc:1428] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 02:14:16.217629 4916 tablet_replica.cc:439] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } }
I20260926 02:14:16.218089 4916 tablet_replica.cc:442] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:14:16.421025 4916 raft_consensus.cc:482] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:14:16.421267 4916 raft_consensus.cc:504] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.422709 4916 leader_election.cc:288] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143), fd011b7327ef47af8963cb038572419d (127.28.152.131:36181)
I20260926 02:14:16.431106 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.423035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:36181 (local address 127.28.152.129:59425)
0926 02:14:16.423631 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:16.423655 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:16.423829 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:16.424577 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:16.424582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:16.424613 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:16.424906 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.424919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.426575 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.426579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.428932 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.428946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.430255 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.430261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.430432 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.430464 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:14:16.430563 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 02:14:16.430707 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":106,"thread_start_us":540,"threads_started":1}
I20260926 02:14:16.431133 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.423199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:59425 (local address 127.28.152.131:36181)
0926 02:14:16.424254 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.424260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.424279 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:14:16.424350 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.424355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.424427 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.424529 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:16.425152 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.426383 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.429233 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.430059 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.430214 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.430290 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:16.430666 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 02:14:16.430819 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":860,"threads_started":1}
I20260926 02:14:16.431527 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.423224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:57115 (local address 127.28.152.130:33143)
0926 02:14:16.423408 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.423415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.424603 (+ 1188us) server_negotiation.cc:407] Connection header received
0926 02:14:16.424850 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.424854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.424913 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.424984 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:16.425557 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.426371 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.429105 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.430322 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.430499 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.430577 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:16.430904 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 02:14:16.431049 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 02:14:16.432229 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.423043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33143 (local address 127.28.152.129:57115)
0926 02:14:16.424503 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:16.424521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:16.424694 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:16.425025 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:16.425030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:16.425052 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:16.425409 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.425418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.426498 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.426503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.428930 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.428946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.430536 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.430543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.430718 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.430746 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 02:14:16.431750 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0926 02:14:16.431894 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"mutex_wait_us":522,"thread_start_us":794,"threads_started":1}
I20260926 02:14:16.432221 4840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fad187480440c3b95e8863087286a9" candidate_uuid: "a110477956164371a2de473051ebb4bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd011b7327ef47af8963cb038572419d" is_pre_election: true
I20260926 02:14:16.432489 4840 raft_consensus.cc:2454] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a110477956164371a2de473051ebb4bb in term 0.
I20260926 02:14:16.432799 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fad187480440c3b95e8863087286a9" candidate_uuid: "a110477956164371a2de473051ebb4bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4f57b618064644b35f926b1d584e5d" is_pre_election: true
I20260926 02:14:16.433008 4695 raft_consensus.cc:2454] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a110477956164371a2de473051ebb4bb in term 0.
I20260926 02:14:16.433014 4489 leader_election.cc:302] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a110477956164371a2de473051ebb4bb, fd011b7327ef47af8963cb038572419d; no voters:
I20260926 02:14:16.433486 4916 raft_consensus.cc:2781] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:14:16.433607 4916 raft_consensus.cc:482] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:14:16.433693 4916 raft_consensus.cc:3037] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:16.436689 4916 raft_consensus.cc:504] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.437554 4916 leader_election.cc:288] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [CANDIDATE]: Term 1 election: Requested vote from peers 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143), fd011b7327ef47af8963cb038572419d (127.28.152.131:36181)
I20260926 02:14:16.438023 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fad187480440c3b95e8863087286a9" candidate_uuid: "a110477956164371a2de473051ebb4bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4f57b618064644b35f926b1d584e5d"
I20260926 02:14:16.438051 4840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fad187480440c3b95e8863087286a9" candidate_uuid: "a110477956164371a2de473051ebb4bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd011b7327ef47af8963cb038572419d"
I20260926 02:14:16.438268 4695 raft_consensus.cc:3037] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:16.438253 4840 raft_consensus.cc:3037] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:16.441169 4695 raft_consensus.cc:2454] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a110477956164371a2de473051ebb4bb in term 1.
I20260926 02:14:16.441315 4840 raft_consensus.cc:2454] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a110477956164371a2de473051ebb4bb in term 1.
I20260926 02:14:16.441738 4489 leader_election.cc:302] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9b4f57b618064644b35f926b1d584e5d, a110477956164371a2de473051ebb4bb, fd011b7327ef47af8963cb038572419d; no voters:
I20260926 02:14:16.442116 4916 raft_consensus.cc:2781] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:16.442579 4916 raft_consensus.cc:686] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [term 1 LEADER]: Becoming Leader. State: Replica: a110477956164371a2de473051ebb4bb, State: Running, Role: LEADER
I20260926 02:14:16.443109 4916 consensus_queue.cc:237] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.445233 4955 tablet_replica.cc:439] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: TxnStatusTablet state changed. Reason: New leader a110477956164371a2de473051ebb4bb. Latest consensus state: current_term: 1 leader_uuid: "a110477956164371a2de473051ebb4bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } }
I20260926 02:14:16.445634 4955 tablet_replica.cc:442] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:14:16.446566 4957 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:14:16.446789 4957 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:14:16.446935 4399 catalog_manager.cc:5495] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb reported cstate change: term changed from 0 to 1, leader changed from <none> to a110477956164371a2de473051ebb4bb (127.28.152.129). New cstate: current_term: 1 leader_uuid: "a110477956164371a2de473051ebb4bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } health_report { overall_health: UNKNOWN } } }
I20260926 02:14:16.509272 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.502955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:39577 (local address 127.0.0.1:44990)
0926 02:14:16.503205 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:16.503225 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:16.503356 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:16.503837 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:16.503840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:16.503849 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:14:16.504001 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.504008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.505427 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.505431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.506449 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.506462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.507470 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.507495 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.507675 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.507739 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:14:16.508773 (+ 1034us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:14:16.508791 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 02:14:16.508849 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 02:14:16.508977 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 02:14:16.509330 4908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.502974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.28.152.129:39577)
0926 02:14:16.503161 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.503167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.503293 (+ 126us) server_negotiation.cc:407] Connection header received
0926 02:14:16.503549 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.503555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.503676 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.503834 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:14:16.504126 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.505286 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.506616 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.507325 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.507465 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.507915 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:14:16.508671 (+ 756us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:14:16.508832 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:16.508881 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 02:14:16.508972 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:14:16.514215 4840 raft_consensus.cc:1261] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Refusing update from remote peer a110477956164371a2de473051ebb4bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:16.514777 4695 raft_consensus.cc:1261] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Refusing update from remote peer a110477956164371a2de473051ebb4bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:16.514904 4955 consensus_queue.cc:1035] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:16.515509 4955 consensus_queue.cc:1035] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:16.525118 4915 tablet_replica.cc:439] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d: TxnStatusTablet state changed. Reason: New leader a110477956164371a2de473051ebb4bb. Latest consensus state: current_term: 1 leader_uuid: "a110477956164371a2de473051ebb4bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } }
I20260926 02:14:16.525524 4915 tablet_replica.cc:442] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:14:16.528474 4960 mvcc.cc:205] Tried to move back new op lower bound from 7333432756275195904 to 7333432755994689536. Current Snapshot: MvccSnapshot[applied={T|T < 7333432756275195904}]
I20260926 02:14:16.531850 4915 tablet_replica.cc:439] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a110477956164371a2de473051ebb4bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } }
I20260926 02:14:16.532210 4915 tablet_replica.cc:442] T b5fad187480440c3b95e8863087286a9 P fd011b7327ef47af8963cb038572419d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:14:16.537411 4955 tablet_replica.cc:439] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a110477956164371a2de473051ebb4bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } }
I20260926 02:14:16.537709 4955 tablet_replica.cc:442] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:14:16.539415 4917 tablet_replica.cc:439] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d: TxnStatusTablet state changed. Reason: New leader a110477956164371a2de473051ebb4bb. Latest consensus state: current_term: 1 leader_uuid: "a110477956164371a2de473051ebb4bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } }
I20260926 02:14:16.539716 4917 tablet_replica.cc:442] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:14:16.542135 4916 tablet_replica.cc:439] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a110477956164371a2de473051ebb4bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } }
I20260926 02:14:16.542440 4916 tablet_replica.cc:442] T b5fad187480440c3b95e8863087286a9 P a110477956164371a2de473051ebb4bb: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:14:16.553828 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.544167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.28.152.131:36181)
0926 02:14:16.544375 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.544382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.545750 (+ 1368us) server_negotiation.cc:407] Connection header received
0926 02:14:16.546098 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.546103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.546214 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.546388 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:14:16.547102 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.548251 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.550606 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.551282 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.551394 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.552125 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:14:16.552961 (+ 836us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:14:16.553094 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:16.553395 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 02:14:16.553505 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:14:16.556041 4917 tablet_replica.cc:439] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a110477956164371a2de473051ebb4bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } } }
I20260926 02:14:16.556330 4917 tablet_replica.cc:442] T b5fad187480440c3b95e8863087286a9 P 9b4f57b618064644b35f926b1d584e5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:14:16.571138 4908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.565821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:36953 (local address 127.28.152.129:39577)
0926 02:14:16.565989 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.565995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.566537 (+ 542us) server_negotiation.cc:407] Connection header received
0926 02:14:16.566714 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.566718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.566789 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.566866 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:14:16.567301 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.568055 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.569381 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.569894 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.570000 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.570391 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:14:16.570520 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:14:16.570592 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:16.570722 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 02:14:16.570801 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 02:14:16.571147 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.565799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:39577 (local address 127.28.152.131:36953)
0926 02:14:16.566417 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:16.566436 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:16.566586 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:16.566902 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:16.566906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:16.566917 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:14:16.567148 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.567156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.568203 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.568208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.569251 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.569263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.570044 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.570050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.570179 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.570234 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:14:16.570608 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:14:16.570620 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:14:16.570688 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 02:14:16.570823 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":300,"threads_started":1}
I20260926 02:14:16.581754 4557 raft_consensus.cc:1261] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Refusing update from remote peer fd011b7327ef47af8963cb038572419d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:16.582406 4695 raft_consensus.cc:1261] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Refusing update from remote peer fd011b7327ef47af8963cb038572419d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:16.582432 4915 consensus_queue.cc:1035] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:16.582999 4915 consensus_queue.cc:1035] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:16.629393 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.622643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:51785 (local address 127.28.152.131:36181)
0926 02:14:16.622852 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.622858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.623484 (+ 626us) server_negotiation.cc:407] Connection header received
0926 02:14:16.623685 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.623691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.623771 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.623854 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:14:16.624456 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.625315 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.627523 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.628008 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.628132 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.628627 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:14:16.628768 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:14:16.628866 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:16.628997 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 02:14:16.629081 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:14:16.629453 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.622539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:36181 (local address 127.28.152.129:51785)
0926 02:14:16.623357 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:16.623378 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:16.623543 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:16.623887 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:16.623892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:16.623903 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:14:16.624265 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.624278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.625426 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.625431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.627284 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:16.627301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.628191 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.628198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:16.628363 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.628444 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:14:16.628869 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:14:16.628879 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:14:16.628965 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 02:14:16.629078 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":493,"threads_started":1}
I20260926 02:14:16.669523 4399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:16.669804 4399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:16.673445 4399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe: Sending DeleteTablet for 3 replicas of tablet fba7a556f7924e24a8a1a97aead98827
I20260926 02:14:16.674515 4676 tablet_service.cc:1514] Processing DeleteTablet for tablet fba7a556f7924e24a8a1a97aead98827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:16 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:55030
I20260926 02:14:16.674540 4537 tablet_service.cc:1514] Processing DeleteTablet for tablet fba7a556f7924e24a8a1a97aead98827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:16 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:44982
I20260926 02:14:16.675683 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet fba7a556f7924e24a8a1a97aead98827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:16 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:53980
I20260926 02:14:16.687158 4982 tablet_replica.cc:330] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d: stopping tablet replica
I20260926 02:14:16.687390 4982 raft_consensus.cc:2229] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:16.687638 4982 raft_consensus.cc:2258] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:16.688032 4983 tablet_replica.cc:330] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb: stopping tablet replica
I20260926 02:14:16.688235 4983 raft_consensus.cc:2229] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:16.688508 4983 raft_consensus.cc:2258] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:16.689739 4982 ts_tablet_manager.cc:1905] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:16.697119 4982 ts_tablet_manager.cc:1918] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:14:16.697304 4982 log.cc:1196] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/fba7a556f7924e24a8a1a97aead98827
I20260926 02:14:16.697736 4982 ts_tablet_manager.cc:1939] T fba7a556f7924e24a8a1a97aead98827 P 9b4f57b618064644b35f926b1d584e5d: Deleting consensus metadata
I20260926 02:14:16.699776 4983 ts_tablet_manager.cc:1905] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:16.700588 4984 tablet_replica.cc:330] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d: stopping tablet replica
I20260926 02:14:16.701283 4984 raft_consensus.cc:2229] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:14:16.702585 4984 raft_consensus.cc:2258] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:16.703665 4984 ts_tablet_manager.cc:1905] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:16.707643 4983 ts_tablet_manager.cc:1918] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:14:16.707798 4983 log.cc:1196] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/fba7a556f7924e24a8a1a97aead98827
I20260926 02:14:16.708205 4983 ts_tablet_manager.cc:1939] T fba7a556f7924e24a8a1a97aead98827 P a110477956164371a2de473051ebb4bb: Deleting consensus metadata
I20260926 02:14:16.709561 4387 catalog_manager.cc:4841] TS a110477956164371a2de473051ebb4bb (127.28.152.129:39577): tablet fba7a556f7924e24a8a1a97aead98827 (table test-table [id=0b2aad7d40aa4c7784c200bbccffe6b7]) successfully deleted
I20260926 02:14:16.710177 4385 catalog_manager.cc:4841] TS 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143): tablet fba7a556f7924e24a8a1a97aead98827 (table test-table [id=0b2aad7d40aa4c7784c200bbccffe6b7]) successfully deleted
I20260926 02:14:16.711390 4984 ts_tablet_manager.cc:1918] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:14:16.711608 4984 log.cc:1196] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/fba7a556f7924e24a8a1a97aead98827
I20260926 02:14:16.712138 4984 ts_tablet_manager.cc:1939] T fba7a556f7924e24a8a1a97aead98827 P fd011b7327ef47af8963cb038572419d: Deleting consensus metadata
I20260926 02:14:16.713388 4386 catalog_manager.cc:4841] TS fd011b7327ef47af8963cb038572419d (127.28.152.131:36181): tablet fba7a556f7924e24a8a1a97aead98827 (table test-table [id=0b2aad7d40aa4c7784c200bbccffe6b7]) successfully deleted
Sep 26 02:14:16 dist-test-slave-hllj krb5kdc[4355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:14:16 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:16 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:14:16.782464 4991 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:16.753315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.28.152.190:33767)
0926 02:14:16.768866 (+ 15551us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.768873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.768900 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:14:16.768965 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.768970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.769049 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.769411 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:16.771355 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.772591 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.774554 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.775272 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.775405 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.782134 (+ 6729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37490: BlockingRecv error: recv got EOF from 127.0.0.1:37490 (error 108)
Metrics: {"server-negotiator.queue_time_us":15398,"thread_start_us":15875,"threads_started":1}
Sep 26 02:14:16 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM
I20260926 02:14:16.814401 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.794575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.28.152.190:33767)
0926 02:14:16.794758 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.794765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.795320 (+ 555us) server_negotiation.cc:407] Connection header received
0926 02:14:16.795508 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.795515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.795603 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.795721 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:16.797458 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.798670 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.800717 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.801435 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.801581 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.806950 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:16.806959 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:16.806964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:16.807017 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:16.810840 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:16.811870 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:16.811877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:16.811881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:16.812111 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:16.812618 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:16.812624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:16.812628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:16.813020 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:16.813291 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:16.813787 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 02:14:16.814038 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:14:16.818971 4399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:14:16.819790 4399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:14:16.827852 4537 tablet_service.cc:1467] Processing CreateTablet for tablet 9409cce1540b4ff29e737d68456d17ba (DEFAULT_TABLE table=test-table [id=4ea4009905e84b3e8ef5824a8458e102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:16.828217 4676 tablet_service.cc:1467] Processing CreateTablet for tablet 9409cce1540b4ff29e737d68456d17ba (DEFAULT_TABLE table=test-table [id=4ea4009905e84b3e8ef5824a8458e102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:16.828365 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9409cce1540b4ff29e737d68456d17ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:16.828615 4676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9409cce1540b4ff29e737d68456d17ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:16.833762 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 9409cce1540b4ff29e737d68456d17ba (DEFAULT_TABLE table=test-table [id=4ea4009905e84b3e8ef5824a8458e102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:16.834257 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9409cce1540b4ff29e737d68456d17ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:16.844722 4998 tablet_bootstrap.cc:492] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb: Bootstrap starting.
I20260926 02:14:16.854332 4999 tablet_bootstrap.cc:492] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d: Bootstrap starting.
I20260926 02:14:16.855322 4998 tablet_bootstrap.cc:654] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:16.856484 5000 tablet_bootstrap.cc:492] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d: Bootstrap starting.
I20260926 02:14:16.858611 5000 tablet_bootstrap.cc:654] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:16.858963 4998 tablet_bootstrap.cc:492] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb: No bootstrap required, opened a new log
I20260926 02:14:16.859133 4998 ts_tablet_manager.cc:1397] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260926 02:14:16.860035 4998 raft_consensus.cc:348] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.860319 4998 raft_consensus.cc:374] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:16.860441 4998 raft_consensus.cc:729] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a110477956164371a2de473051ebb4bb, State: Initialized, Role: FOLLOWER
I20260926 02:14:16.860884 4998 consensus_queue.cc:260] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.865563 4998 ts_tablet_manager.cc:1428] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260926 02:14:16.867622 5000 tablet_bootstrap.cc:492] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d: No bootstrap required, opened a new log
I20260926 02:14:16.867849 5000 ts_tablet_manager.cc:1397] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260926 02:14:16.868700 5000 raft_consensus.cc:348] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.869074 5000 raft_consensus.cc:374] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:16.869246 5000 raft_consensus.cc:729] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4f57b618064644b35f926b1d584e5d, State: Initialized, Role: FOLLOWER
I20260926 02:14:16.869364 4999 tablet_bootstrap.cc:654] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:16.869613 5000 consensus_queue.cc:260] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.872658 5000 ts_tablet_manager.cc:1428] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 02:14:16.880837 4999 tablet_bootstrap.cc:492] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d: No bootstrap required, opened a new log
I20260926 02:14:16.881023 4999 ts_tablet_manager.cc:1397] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.005s
I20260926 02:14:16.881832 4999 raft_consensus.cc:348] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.882099 4999 raft_consensus.cc:374] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:16.882205 4999 raft_consensus.cc:729] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd011b7327ef47af8963cb038572419d, State: Initialized, Role: FOLLOWER
I20260926 02:14:16.882483 4999 consensus_queue.cc:260] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.883170 4999 ts_tablet_manager.cc:1428] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 02:14:16.943979 4915 raft_consensus.cc:482] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:14:16.944196 4915 raft_consensus.cc:504] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.944832 4915 leader_election.cc:288] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a110477956164371a2de473051ebb4bb (127.28.152.129:39577), 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143)
I20260926 02:14:16.945478 4557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9409cce1540b4ff29e737d68456d17ba" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a110477956164371a2de473051ebb4bb" is_pre_election: true
I20260926 02:14:16.945540 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9409cce1540b4ff29e737d68456d17ba" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4f57b618064644b35f926b1d584e5d" is_pre_election: true
I20260926 02:14:16.945744 4695 raft_consensus.cc:2454] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 0.
I20260926 02:14:16.945740 4557 raft_consensus.cc:2454] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 0.
I20260926 02:14:16.946237 4767 leader_election.cc:302] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b4f57b618064644b35f926b1d584e5d, fd011b7327ef47af8963cb038572419d; no voters:
I20260926 02:14:16.946677 4915 raft_consensus.cc:2781] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:14:16.946779 4915 raft_consensus.cc:482] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:14:16.946889 4915 raft_consensus.cc:3037] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:16.949429 4915 raft_consensus.cc:504] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.950438 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9409cce1540b4ff29e737d68456d17ba" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4f57b618064644b35f926b1d584e5d"
I20260926 02:14:16.950459 4557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9409cce1540b4ff29e737d68456d17ba" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a110477956164371a2de473051ebb4bb"
I20260926 02:14:16.950577 4915 leader_election.cc:288] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 election: Requested vote from peers a110477956164371a2de473051ebb4bb (127.28.152.129:39577), 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143)
I20260926 02:14:16.950654 4695 raft_consensus.cc:3037] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:16.950654 4557 raft_consensus.cc:3037] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:16.953485 4557 raft_consensus.cc:2454] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 1.
I20260926 02:14:16.953485 4695 raft_consensus.cc:2454] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 1.
I20260926 02:14:16.953969 4769 leader_election.cc:302] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a110477956164371a2de473051ebb4bb, fd011b7327ef47af8963cb038572419d; no voters:
I20260926 02:14:16.954231 4915 raft_consensus.cc:2781] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:16.954389 4915 raft_consensus.cc:686] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 1 LEADER]: Becoming Leader. State: Replica: fd011b7327ef47af8963cb038572419d, State: Running, Role: LEADER
I20260926 02:14:16.954624 4915 consensus_queue.cc:237] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:16.957468 4398 catalog_manager.cc:5495] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d reported cstate change: term changed from 0 to 1, leader changed from <none> to fd011b7327ef47af8963cb038572419d (127.28.152.131). New cstate: current_term: 1 leader_uuid: "fd011b7327ef47af8963cb038572419d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } health_report { overall_health: HEALTHY } } }
I20260926 02:14:16.982724 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:16.977263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.28.152.131:36181)
0926 02:14:16.977478 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:16.977485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:16.977507 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:14:16.977587 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:16.977592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:16.977682 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:16.977783 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:14:16.978164 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.979335 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.980723 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:16.981454 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:16.981591 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:16.982000 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:14:16.982150 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:14:16.982238 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:16.982354 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 02:14:16.982436 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:14:16.985710 4695 raft_consensus.cc:1261] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Refusing update from remote peer fd011b7327ef47af8963cb038572419d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:16.985698 4557 raft_consensus.cc:1261] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Refusing update from remote peer fd011b7327ef47af8963cb038572419d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:16.986367 4974 consensus_queue.cc:1035] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:16.986680 4915 consensus_queue.cc:1035] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:16.992944 4963 mvcc.cc:205] Tried to move back new op lower bound from 7333432758207152128 to 7333432758088413184. Current Snapshot: MvccSnapshot[applied={T|T < 7333432758207152128}]
I20260926 02:14:16.998941 4399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:16.999217 4399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:17.002084 4399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe: Sending DeleteTablet for 3 replicas of tablet 9409cce1540b4ff29e737d68456d17ba
I20260926 02:14:17.002938 4537 tablet_service.cc:1514] Processing DeleteTablet for tablet 9409cce1540b4ff29e737d68456d17ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:16 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:44982
I20260926 02:14:17.002928 4676 tablet_service.cc:1514] Processing DeleteTablet for tablet 9409cce1540b4ff29e737d68456d17ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:16 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:55030
I20260926 02:14:17.002916 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 9409cce1540b4ff29e737d68456d17ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:16 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:53980
I20260926 02:14:17.003207 4983 tablet_replica.cc:330] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb: stopping tablet replica
I20260926 02:14:17.003212 4984 tablet_replica.cc:330] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d: stopping tablet replica
I20260926 02:14:17.003387 4984 raft_consensus.cc:2229] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:14:17.003397 4983 raft_consensus.cc:2229] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:17.003589 4983 raft_consensus.cc:2258] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:17.003662 4984 raft_consensus.cc:2258] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:17.003837 4982 tablet_replica.cc:330] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d: stopping tablet replica
I20260926 02:14:17.004030 4982 raft_consensus.cc:2229] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:17.004252 4982 raft_consensus.cc:2258] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:17.004655 4983 ts_tablet_manager.cc:1905] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:17.005112 4984 ts_tablet_manager.cc:1905] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:17.005803 4982 ts_tablet_manager.cc:1905] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:17.008579 4398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:14:17.009317 4398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:14:17.011822 4983 ts_tablet_manager.cc:1918] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:17.012025 4983 log.cc:1196] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/9409cce1540b4ff29e737d68456d17ba
I20260926 02:14:17.012378 4982 ts_tablet_manager.cc:1918] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:17.012547 4982 log.cc:1196] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/9409cce1540b4ff29e737d68456d17ba
I20260926 02:14:17.012704 4983 ts_tablet_manager.cc:1939] T 9409cce1540b4ff29e737d68456d17ba P a110477956164371a2de473051ebb4bb: Deleting consensus metadata
I20260926 02:14:17.012928 4982 ts_tablet_manager.cc:1939] T 9409cce1540b4ff29e737d68456d17ba P 9b4f57b618064644b35f926b1d584e5d: Deleting consensus metadata
I20260926 02:14:17.014089 4385 catalog_manager.cc:4841] TS 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143): tablet 9409cce1540b4ff29e737d68456d17ba (table test-table [id=4ea4009905e84b3e8ef5824a8458e102]) successfully deleted
I20260926 02:14:17.014387 4387 catalog_manager.cc:4841] TS a110477956164371a2de473051ebb4bb (127.28.152.129:39577): tablet 9409cce1540b4ff29e737d68456d17ba (table test-table [id=4ea4009905e84b3e8ef5824a8458e102]) successfully deleted
I20260926 02:14:17.017939 4537 tablet_service.cc:1467] Processing CreateTablet for tablet c7c5988245304402b826e5c6d004f16c (DEFAULT_TABLE table=test-table [id=1aa27600e2414139bb5fbe26092de6df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:17.018006 4676 tablet_service.cc:1467] Processing CreateTablet for tablet c7c5988245304402b826e5c6d004f16c (DEFAULT_TABLE table=test-table [id=1aa27600e2414139bb5fbe26092de6df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:17.018177 4818 tablet_service.cc:1467] Processing CreateTablet for tablet c7c5988245304402b826e5c6d004f16c (DEFAULT_TABLE table=test-table [id=1aa27600e2414139bb5fbe26092de6df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:17.018396 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c5988245304402b826e5c6d004f16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:17.018426 4676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c5988245304402b826e5c6d004f16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:17.018590 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c5988245304402b826e5c6d004f16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:17.024638 5000 tablet_bootstrap.cc:492] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d: Bootstrap starting.
I20260926 02:14:17.025031 4999 tablet_bootstrap.cc:492] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d: Bootstrap starting.
I20260926 02:14:17.026314 5000 tablet_bootstrap.cc:654] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:17.026772 4999 tablet_bootstrap.cc:654] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:17.031126 4998 tablet_bootstrap.cc:492] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb: Bootstrap starting.
I20260926 02:14:17.033046 4998 tablet_bootstrap.cc:654] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:17.036711 5000 tablet_bootstrap.cc:492] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d: No bootstrap required, opened a new log
I20260926 02:14:17.036947 5000 ts_tablet_manager.cc:1397] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.002s
I20260926 02:14:17.037837 5000 raft_consensus.cc:348] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:17.038456 5000 raft_consensus.cc:374] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:17.038558 5000 raft_consensus.cc:729] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4f57b618064644b35f926b1d584e5d, State: Initialized, Role: FOLLOWER
I20260926 02:14:17.038931 5000 consensus_queue.cc:260] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:17.029517 4984 ts_tablet_manager.cc:1918] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:17.040005 4984 log.cc:1196] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/9409cce1540b4ff29e737d68456d17ba
I20260926 02:14:17.040530 4984 ts_tablet_manager.cc:1939] T 9409cce1540b4ff29e737d68456d17ba P fd011b7327ef47af8963cb038572419d: Deleting consensus metadata
I20260926 02:14:17.041832 4386 catalog_manager.cc:4841] TS fd011b7327ef47af8963cb038572419d (127.28.152.131:36181): tablet 9409cce1540b4ff29e737d68456d17ba (table test-table [id=4ea4009905e84b3e8ef5824a8458e102]) successfully deleted
I20260926 02:14:17.042130 5000 ts_tablet_manager.cc:1428] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 02:14:17.046398 4998 tablet_bootstrap.cc:492] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb: No bootstrap required, opened a new log
I20260926 02:14:17.046571 4998 ts_tablet_manager.cc:1397] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.005s
I20260926 02:14:17.047158 4999 tablet_bootstrap.cc:492] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d: No bootstrap required, opened a new log
I20260926 02:14:17.047338 4999 ts_tablet_manager.cc:1397] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.005s
I20260926 02:14:17.047391 4998 raft_consensus.cc:348] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:17.047873 4998 raft_consensus.cc:374] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:17.047961 4998 raft_consensus.cc:729] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a110477956164371a2de473051ebb4bb, State: Initialized, Role: FOLLOWER
I20260926 02:14:17.048116 4999 raft_consensus.cc:348] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:17.048375 4999 raft_consensus.cc:374] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:17.048480 4999 raft_consensus.cc:729] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd011b7327ef47af8963cb038572419d, State: Initialized, Role: FOLLOWER
I20260926 02:14:17.048233 4998 consensus_queue.cc:260] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:17.048758 4999 consensus_queue.cc:260] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:17.049022 4998 ts_tablet_manager.cc:1428] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 02:14:17.050061 4999 ts_tablet_manager.cc:1428] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 02:14:17.141348 4915 raft_consensus.cc:482] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:14:17.141599 4915 raft_consensus.cc:504] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:17.142416 4915 leader_election.cc:288] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a110477956164371a2de473051ebb4bb (127.28.152.129:39577), 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143)
I20260926 02:14:17.142974 4557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c5988245304402b826e5c6d004f16c" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a110477956164371a2de473051ebb4bb" is_pre_election: true
I20260926 02:14:17.143054 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c5988245304402b826e5c6d004f16c" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4f57b618064644b35f926b1d584e5d" is_pre_election: true
I20260926 02:14:17.143206 4557 raft_consensus.cc:2454] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 0.
I20260926 02:14:17.143322 4695 raft_consensus.cc:2454] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 0.
I20260926 02:14:17.143640 4769 leader_election.cc:302] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a110477956164371a2de473051ebb4bb, fd011b7327ef47af8963cb038572419d; no voters:
I20260926 02:14:17.144016 4915 raft_consensus.cc:2781] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:14:17.144125 4915 raft_consensus.cc:482] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:14:17.144224 4915 raft_consensus.cc:3037] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:17.147251 4915 raft_consensus.cc:504] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:17.147979 4915 leader_election.cc:288] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 election: Requested vote from peers a110477956164371a2de473051ebb4bb (127.28.152.129:39577), 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143)
I20260926 02:14:17.148511 4557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c5988245304402b826e5c6d004f16c" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a110477956164371a2de473051ebb4bb"
I20260926 02:14:17.148528 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c5988245304402b826e5c6d004f16c" candidate_uuid: "fd011b7327ef47af8963cb038572419d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4f57b618064644b35f926b1d584e5d"
I20260926 02:14:17.148723 4695 raft_consensus.cc:3037] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:17.148713 4557 raft_consensus.cc:3037] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:17.150763 4557 raft_consensus.cc:2454] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 1.
I20260926 02:14:17.151258 4769 leader_election.cc:302] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a110477956164371a2de473051ebb4bb, fd011b7327ef47af8963cb038572419d; no voters:
I20260926 02:14:17.151520 4915 raft_consensus.cc:2781] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:17.151624 4695 raft_consensus.cc:2454] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd011b7327ef47af8963cb038572419d in term 1.
I20260926 02:14:17.151698 4915 raft_consensus.cc:686] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 1 LEADER]: Becoming Leader. State: Replica: fd011b7327ef47af8963cb038572419d, State: Running, Role: LEADER
I20260926 02:14:17.152215 4915 consensus_queue.cc:237] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } }
I20260926 02:14:17.155071 4399 catalog_manager.cc:5495] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d reported cstate change: term changed from 0 to 1, leader changed from <none> to fd011b7327ef47af8963cb038572419d (127.28.152.131). New cstate: current_term: 1 leader_uuid: "fd011b7327ef47af8963cb038572419d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd011b7327ef47af8963cb038572419d" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 36181 } health_report { overall_health: HEALTHY } } }
I20260926 02:14:17.207293 4695 raft_consensus.cc:1261] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Refusing update from remote peer fd011b7327ef47af8963cb038572419d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:17.207446 4557 raft_consensus.cc:1261] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Refusing update from remote peer fd011b7327ef47af8963cb038572419d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:17.208002 4977 consensus_queue.cc:1035] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b4f57b618064644b35f926b1d584e5d" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:17.208406 4915 consensus_queue.cc:1035] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a110477956164371a2de473051ebb4bb" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:17.253644 4398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:17.253844 4398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:17.256621 4398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21762cb54a984847b4c717886024eafe: Sending DeleteTablet for 3 replicas of tablet c7c5988245304402b826e5c6d004f16c
I20260926 02:14:17.257465 4676 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c5988245304402b826e5c6d004f16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:17 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:55030
I20260926 02:14:17.257442 4537 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c5988245304402b826e5c6d004f16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:17 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:44982
I20260926 02:14:17.257762 4983 tablet_replica.cc:330] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb: stopping tablet replica
I20260926 02:14:17.257824 4982 tablet_replica.cc:330] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d: stopping tablet replica
I20260926 02:14:17.257937 4983 raft_consensus.cc:2229] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:17.258014 4982 raft_consensus.cc:2229] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:17.258134 4983 raft_consensus.cc:2258] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:17.258200 4982 raft_consensus.cc:2258] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:17.258700 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c5988245304402b826e5c6d004f16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:17 UTC) from {username='oryx', principal='oryx/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:53980
I20260926 02:14:17.258850 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 4471
I20260926 02:14:17.259313 4982 ts_tablet_manager.cc:1905] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:17.259343 4983 ts_tablet_manager.cc:1905] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:17.259922 4984 tablet_replica.cc:330] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d: stopping tablet replica
I20260926 02:14:17.260116 4984 raft_consensus.cc:2229] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:14:17.260440 4984 raft_consensus.cc:2258] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:17.261581 4984 ts_tablet_manager.cc:1905] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:17.265879 4982 ts_tablet_manager.cc:1918] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:14:17.266031 4982 log.cc:1196] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/c7c5988245304402b826e5c6d004f16c
I20260926 02:14:17.266228 4983 ts_tablet_manager.cc:1918] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:14:17.266382 4983 log.cc:1196] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/c7c5988245304402b826e5c6d004f16c
I20260926 02:14:17.266464 4982 ts_tablet_manager.cc:1939] T c7c5988245304402b826e5c6d004f16c P 9b4f57b618064644b35f926b1d584e5d: Deleting consensus metadata
I20260926 02:14:17.266829 4983 ts_tablet_manager.cc:1939] T c7c5988245304402b826e5c6d004f16c P a110477956164371a2de473051ebb4bb: Deleting consensus metadata
I20260926 02:14:17.267566 4385 catalog_manager.cc:4841] TS 9b4f57b618064644b35f926b1d584e5d (127.28.152.130:33143): tablet c7c5988245304402b826e5c6d004f16c (table test-table [id=1aa27600e2414139bb5fbe26092de6df]) successfully deleted
I20260926 02:14:17.268313 4387 catalog_manager.cc:4841] TS a110477956164371a2de473051ebb4bb (127.28.152.129:39577): tablet c7c5988245304402b826e5c6d004f16c (table test-table [id=1aa27600e2414139bb5fbe26092de6df]) successfully deleted
I20260926 02:14:17.269025 4984 ts_tablet_manager.cc:1918] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:14:17.269162 4984 log.cc:1196] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/c7c5988245304402b826e5c6d004f16c
I20260926 02:14:17.269479 4984 ts_tablet_manager.cc:1939] T c7c5988245304402b826e5c6d004f16c P fd011b7327ef47af8963cb038572419d: Deleting consensus metadata
I20260926 02:14:17.270313 4386 catalog_manager.cc:4841] TS fd011b7327ef47af8963cb038572419d (127.28.152.131:36181): tablet c7c5988245304402b826e5c6d004f16c (table test-table [id=1aa27600e2414139bb5fbe26092de6df]) successfully deleted
Sep 26 02:14:17 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.152.129@KRBTEST.COM
I20260926 02:14:17.282492 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:17.260337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.28.152.129:39577)
0926 02:14:17.268317 (+ 7980us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:17.268323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:17.268849 (+ 526us) server_negotiation.cc:407] Connection header received
0926 02:14:17.269067 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:17.269072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:17.269169 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:17.269281 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:17.270658 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:17.271338 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:17.272564 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:17.272935 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:17.273041 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:17.276856 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:17.276872 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:17.276889 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:17.276928 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:17.279556 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:17.280299 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:17.280308 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:17.280310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:17.280474 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:17.280877 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:17.280881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:17.280884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:17.281314 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:17.281611 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:17.281932 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 02:14:17.282150 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7850,"thread_start_us":7732,"threads_started":1}
I20260926 02:14:17.283097 4602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:17.404263 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 4471
I20260926 02:14:17.423219 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 4611
Sep 26 02:14:17 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.152.130@KRBTEST.COM
I20260926 02:14:17.439563 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:17.424545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.28.152.130:33143)
0926 02:14:17.424965 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:17.424969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:17.424985 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:14:17.425038 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:17.425042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:17.425100 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:17.425235 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:17.426519 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:17.427353 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:17.428763 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:17.429275 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:17.429387 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:17.433634 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:17.433652 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:17.433670 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:17.433709 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:17.436696 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:17.437502 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:17.437511 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:17.437513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:17.437646 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:17.438014 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:17.438020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:17.438022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:17.438393 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:17.438695 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:17.438995 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 02:14:17.439213 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":232,"threads_started":1}
I20260926 02:14:17.440160 4741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:17.562904 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 4611
I20260926 02:14:17.581600 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 4750
Sep 26 02:14:17 dist-test-slave-hllj krb5kdc[4355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.152.131@KRBTEST.COM
I20260926 02:14:17.595948 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:17.582308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.28.152.131:36181)
0926 02:14:17.582741 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:17.582745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:17.582761 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:14:17.582814 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:17.582818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:17.582879 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:17.582993 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:17.584226 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:17.584925 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:17.586202 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:17.586587 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:17.586680 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:17.590965 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:17.590981 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:17.591023 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:17.591062 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:17.593598 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:17.594225 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:17.594238 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:17.594240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:17.594362 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:17.594628 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:17.594630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:17.594632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:17.594962 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:17.595179 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:17.595379 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 02:14:17.595552 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":249,"threads_started":1}
I20260926 02:14:17.596633 4880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:17.717617 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 4750
I20260926 02:14:17.735534 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 4368
I20260926 02:14:17.736886 4440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:17.851434 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 4368
2026-09-26T02:14:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5756 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:14:17 dist-test-slave-hllj krb5kdc[5029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:14:17 dist-test-slave-hllj krb5kdc[5029](info): set up 2 sockets
Sep 26 02:14:17 dist-test-slave-hllj krb5kdc[5029](info): commencing operation
krb5kdc: starting...
W20260926 02:14:18.875291 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:14:19 dist-test-slave-hllj krb5kdc[5029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:14:19.090102 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:39217
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:39217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:19.419380 5042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:19.419715 5042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:19.419845 5042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:19.427201 5042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:14:19.427366 5042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:19.427448 5042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:19.427529 5042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:14:19.427593 5042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:14:19.437659 5042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:39217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:39217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:19.438992 5042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:19.440384 5042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:19.453433 5048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:19.453377 5047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:19.453343 5050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:19.453939 5042 server_base.cc:1034] running on GCE node
I20260926 02:14:19.454911 5042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:19.456455 5042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:19.457674 5042 hybrid_clock.cc:648] HybridClock initialized: now 1790388859457623 us; error 50 us; skew 500 ppm
I20260926 02:14:19.458031 5042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T02:14:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1669 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:14:19 dist-test-slave-hllj krb5kdc[5056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:14:19 dist-test-slave-hllj krb5kdc[5056](info): set up 2 sockets
Sep 26 02:14:19 dist-test-slave-hllj krb5kdc[5056](info): commencing operation
krb5kdc: starting...
W20260926 02:14:20.535425 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:14:20 dist-test-slave-hllj krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:14:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:14:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:14:20.753916 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:36985
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:36985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:21.085180 5069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:21.085513 5069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:21.085634 5069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:21.093116 5069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:14:21.093310 5069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:21.093390 5069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:21.093469 5069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:14:21.093531 5069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:14:21.103650 5069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:36985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:36985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:21.104979 5069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:21.106302 5069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:21.117811 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:21.117782 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:21.117753 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:21.118485 5069 server_base.cc:1034] running on GCE node
I20260926 02:14:21.119256 5069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:21.120707 5069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:21.121907 5069 hybrid_clock.cc:648] HybridClock initialized: now 1790388861121842 us; error 56 us; skew 500 ppm
I20260926 02:14:21.122259 5069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:21 dist-test-slave-hllj krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388861, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:21.125283 5069 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:14:21.126782 5069 webserver.cc:460] Webserver started at http://127.28.152.190:44559/ using document root <none> and password file <none>
I20260926 02:14:21.127487 5069 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:21.127600 5069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:21.127898 5069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:21.130879 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "9f17858b9b874e8f81fae45e0fac9546"
format_stamp: "Formatted at 2026-09-26 02:14:21 on dist-test-slave-hllj"
server_key: "95f558f88c3f1afe832c5265ae03d5cf"
server_key_iv: "6f76cd33225d1381de302601c5b72239"
server_key_version: "encryptionkey@0"
I20260926 02:14:21.131632 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "9f17858b9b874e8f81fae45e0fac9546"
format_stamp: "Formatted at 2026-09-26 02:14:21 on dist-test-slave-hllj"
server_key: "95f558f88c3f1afe832c5265ae03d5cf"
server_key_iv: "6f76cd33225d1381de302601c5b72239"
server_key_version: "encryptionkey@0"
I20260926 02:14:21.136824 5069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:14:21.140599 5084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:21.142161 5069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:14:21.142359 5069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "9f17858b9b874e8f81fae45e0fac9546"
format_stamp: "Formatted at 2026-09-26 02:14:21 on dist-test-slave-hllj"
server_key: "95f558f88c3f1afe832c5265ae03d5cf"
server_key_iv: "6f76cd33225d1381de302601c5b72239"
server_key_version: "encryptionkey@0"
I20260926 02:14:21.142556 5069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:21.156200 5069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:21.160774 5069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:21.161072 5069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:21.178516 5069 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:36985
I20260926 02:14:21.178517 5135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:36985 every 8 connection(s)
I20260926 02:14:21.179939 5069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:14:21.183825 5136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:21.185854 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 5069
I20260926 02:14:21.186082 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:14:21.186687 29282 external_mini_cluster.cc:1428] Setting key bfdf72d2a61530d4a906784f8429ffe5
I20260926 02:14:21.196775 5136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546: Bootstrap starting.
I20260926 02:14:21.200466 5136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:21.201581 5136 log.cc:824] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:14:21 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:21.204783 5136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546: No bootstrap required, opened a new log
I20260926 02:14:21.211674 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:21.188796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.28.152.190:36985)
0926 02:14:21.189625 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:21.189642 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:21.189694 (+ 52us) server_negotiation.cc:407] Connection header received
0926 02:14:21.191970 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:21.192002 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:21.192555 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:21.193066 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:21.194476 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:21.196410 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:21.197974 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:21.199130 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:21.199324 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:21.203553 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:21.203563 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:21.203581 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:21.203619 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:21.206979 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:21.207964 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:21.207971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:21.207979 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:21.208172 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:21.208665 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:21.208669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:21.208671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:21.209343 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:21.209617 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:21.210073 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 02:14:21.210481 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":364,"threads_started":1}
I20260926 02:14:21.214144 5136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } }
I20260926 02:14:21.214545 5136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:21.214636 5136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f17858b9b874e8f81fae45e0fac9546, State: Initialized, Role: FOLLOWER
I20260926 02:14:21.215178 5136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } }
I20260926 02:14:21.215404 5136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:14:21.215528 5136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:14:21.215678 5136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:21.217887 5136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } }
I20260926 02:14:21.218411 5136 leader_election.cc:302] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f17858b9b874e8f81fae45e0fac9546; no voters:
I20260926 02:14:21.218950 5136 leader_election.cc:288] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:14:21.219076 5141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:21.219696 5141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 1 LEADER]: Becoming Leader. State: Replica: 9f17858b9b874e8f81fae45e0fac9546, State: Running, Role: LEADER
I20260926 02:14:21.220223 5141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } }
I20260926 02:14:21.220914 5136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:14:21.222945 5143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f17858b9b874e8f81fae45e0fac9546. Latest consensus state: current_term: 1 leader_uuid: "9f17858b9b874e8f81fae45e0fac9546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } } }
I20260926 02:14:21.222877 5142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f17858b9b874e8f81fae45e0fac9546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } } }
I20260926 02:14:21.223289 5142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:21.223294 5143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:21.226719 5147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:14:21.231714 5147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:14:21.241825 5147 catalog_manager.cc:1347] Generated new cluster ID: 184c7460bd6b4964b880624c9b91a8fb
I20260926 02:14:21.241941 5147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:14:21.252307 5147 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:14:21.253724 5147 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:14:21.270161 5147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546: Generated new TSK 0
I20260926 02:14:21.271111 5147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:14:21.325752 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36985
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:21.664865 5162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:21.665246 5162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:21.665531 5162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:21.673070 5162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:21.673247 5162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:21.673405 5162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:14:21.683411 5162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:21.684772 5162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:21.686228 5162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:21.699290 5167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:21.699365 5168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:21.699394 5170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:21.700088 5162 server_base.cc:1034] running on GCE node
I20260926 02:14:21.700678 5162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:21.701678 5162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:21.702939 5162 hybrid_clock.cc:648] HybridClock initialized: now 1790388861702859 us; error 58 us; skew 500 ppm
I20260926 02:14:21.703272 5162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:21 dist-test-slave-hllj krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388861, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:21.706259 5162 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:14:21.707708 5162 webserver.cc:460] Webserver started at http://127.28.152.129:45595/ using document root <none> and password file <none>
I20260926 02:14:21.708398 5162 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:21.708513 5162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:21.708804 5162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:21.711757 5162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "b413609c89634a228daa6a6a5a989483"
format_stamp: "Formatted at 2026-09-26 02:14:21 on dist-test-slave-hllj"
server_key: "55e88b26bec8abda12c48acf6d2d82b5"
server_key_iv: "735f6f303d144926af2e948912407691"
server_key_version: "encryptionkey@0"
I20260926 02:14:21.712589 5162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "b413609c89634a228daa6a6a5a989483"
format_stamp: "Formatted at 2026-09-26 02:14:21 on dist-test-slave-hllj"
server_key: "55e88b26bec8abda12c48acf6d2d82b5"
server_key_iv: "735f6f303d144926af2e948912407691"
server_key_version: "encryptionkey@0"
I20260926 02:14:21.717913 5162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 02:14:21.721333 5177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:21.722815 5162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:14:21.722972 5162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "b413609c89634a228daa6a6a5a989483"
format_stamp: "Formatted at 2026-09-26 02:14:21 on dist-test-slave-hllj"
server_key: "55e88b26bec8abda12c48acf6d2d82b5"
server_key_iv: "735f6f303d144926af2e948912407691"
server_key_version: "encryptionkey@0"
I20260926 02:14:21.723160 5162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:21.742015 5162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:21.746104 5162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:21.746397 5162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:21.747377 5162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:21.748757 5162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:21.748836 5162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:21.748961 5162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:21.749017 5162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:21.778909 5162 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:37793
I20260926 02:14:21.778956 5289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:37793 every 8 connection(s)
I20260926 02:14:21.780184 5162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:14:21.787827 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 5162
I20260926 02:14:21.788064 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:14:21.788740 29282 external_mini_cluster.cc:1428] Setting key 7fc2a10c94e281f038eea0e54707a89f
Sep 26 02:14:21 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388861, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:21.802030 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:21.782736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:56235 (local address 127.28.152.190:36985)
0926 02:14:21.783166 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:21.783172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:21.784243 (+ 1071us) server_negotiation.cc:407] Connection header received
0926 02:14:21.786164 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:21.786168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:21.786239 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:21.786352 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:21.788328 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:21.789324 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:21.791433 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:21.792009 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:21.792167 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:21.795580 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:21.795587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:21.795591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:21.795628 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:21.798665 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:21.799617 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:21.799622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:21.799624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:21.799741 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:21.800250 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:21.800254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:21.800256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:21.800558 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:21.800680 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:21.801581 (+ 901us) server_negotiation.cc:299] Negotiation successful
0926 02:14:21.801773 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":203,"threads_started":1}
I20260926 02:14:21.803458 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:21.783311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:56235)
0926 02:14:21.784085 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:21.784123 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:21.785877 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:21.786626 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:21.786635 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:21.787179 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:21.788122 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:21.788150 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:21.789565 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:21.789571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:21.791278 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:21.791291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:21.792263 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:21.792269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:21.792507 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:21.793168 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:21.793202 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:21.795334 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:21.798836 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:21.798847 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:21.798868 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:21.799501 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:21.799834 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:21.799840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:21.799843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:21.800110 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:21.800686 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:21.800694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:21.801169 (+ 475us) client_negotiation.cc:769] Sending connection context
0926 02:14:21.801475 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 02:14:21.802057 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":304,"threads_started":1}
I20260926 02:14:21.805120 5290 heartbeater.cc:347] Connected to a master server at 127.28.152.190:36985
I20260926 02:14:21.805462 5290 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:21.806228 5290 heartbeater.cc:511] Master 127.28.152.190:36985 requested a full tablet report, sending...
I20260926 02:14:21.808413 5101 ts_manager.cc:194] Registered new tserver with Master: b413609c89634a228daa6a6a5a989483 (127.28.152.129:37793)
I20260926 02:14:21.809917 5101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:56235
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:14:21.840404 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36985
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:22.170485 5296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:22.170790 5296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:22.170926 5296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:22.178283 5296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:22.178409 5296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:22.178565 5296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:14:22.188469 5296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:22.189827 5296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:22.191186 5296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:22.204144 5301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:22.204200 5304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:22.204207 5302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:22.204763 5296 server_base.cc:1034] running on GCE node
I20260926 02:14:22.205493 5296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:22.206485 5296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:22.207703 5296 hybrid_clock.cc:648] HybridClock initialized: now 1790388862207648 us; error 39 us; skew 500 ppm
I20260926 02:14:22.208043 5296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:22 dist-test-slave-hllj krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388862, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:22.211140 5296 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:14:22.212662 5296 webserver.cc:460] Webserver started at http://127.28.152.130:46535/ using document root <none> and password file <none>
I20260926 02:14:22.213433 5296 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:22.213544 5296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:22.213843 5296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:22.216749 5296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "1b019d8b753640cc94168a219299e48f"
format_stamp: "Formatted at 2026-09-26 02:14:22 on dist-test-slave-hllj"
server_key: "5b7df5e6e41fdf92a4684d214d98b9d0"
server_key_iv: "6d3f389f650c9c68c91efdf67a9a20c0"
server_key_version: "encryptionkey@0"
I20260926 02:14:22.217527 5296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "1b019d8b753640cc94168a219299e48f"
format_stamp: "Formatted at 2026-09-26 02:14:22 on dist-test-slave-hllj"
server_key: "5b7df5e6e41fdf92a4684d214d98b9d0"
server_key_iv: "6d3f389f650c9c68c91efdf67a9a20c0"
server_key_version: "encryptionkey@0"
I20260926 02:14:22.222776 5296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:14:22.226269 5311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:22.228026 5296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:14:22.228188 5296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "1b019d8b753640cc94168a219299e48f"
format_stamp: "Formatted at 2026-09-26 02:14:22 on dist-test-slave-hllj"
server_key: "5b7df5e6e41fdf92a4684d214d98b9d0"
server_key_iv: "6d3f389f650c9c68c91efdf67a9a20c0"
server_key_version: "encryptionkey@0"
I20260926 02:14:22.228397 5296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:22.244624 5296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:22.248805 5296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:22.249074 5296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:22.250069 5296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:22.251507 5296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:22.251583 5296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:22.251711 5296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:22.251766 5296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:22.282244 5296 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:44119
I20260926 02:14:22.282276 5423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:44119 every 8 connection(s)
I20260926 02:14:22.283596 5296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:14:22.291497 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 5296
I20260926 02:14:22.291683 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:14:22.292085 29282 external_mini_cluster.cc:1428] Setting key 7157dfccce35f5b88e42670b67b293fa
Sep 26 02:14:22 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388862, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:22.304978 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:22.286199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:38831 (local address 127.28.152.190:36985)
0926 02:14:22.286386 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:22.286391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:22.287683 (+ 1292us) server_negotiation.cc:407] Connection header received
0926 02:14:22.289561 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:22.289566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:22.289643 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:22.289733 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:22.291368 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:22.292136 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:22.294081 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:22.294506 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:22.294602 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:22.298678 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:22.298685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:22.298690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:22.298732 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:22.301655 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:22.302644 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:22.302649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:22.302651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:22.302782 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:22.303244 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:22.303248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:22.303250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:22.303544 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:22.303686 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:22.304409 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 02:14:22.304651 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:14:22.306048 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:22.286783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:38831)
0926 02:14:22.287514 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:22.287561 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:22.289266 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:22.289954 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:22.289963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:22.290514 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:22.291157 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:22.291184 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:22.292312 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:22.292317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:22.293947 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:22.293960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:22.294734 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:22.294742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:22.295031 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:22.295832 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:22.295861 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:22.298417 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:22.301816 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:22.301828 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:22.301851 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:22.302515 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:22.302893 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:22.302898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:22.302902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:22.303122 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:22.303668 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:22.303677 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:22.304043 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 02:14:22.304302 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 02:14:22.304888 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":285,"threads_started":1}
I20260926 02:14:22.307703 5424 heartbeater.cc:347] Connected to a master server at 127.28.152.190:36985
I20260926 02:14:22.308014 5424 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:22.308796 5424 heartbeater.cc:511] Master 127.28.152.190:36985 requested a full tablet report, sending...
I20260926 02:14:22.310475 5101 ts_manager.cc:194] Registered new tserver with Master: 1b019d8b753640cc94168a219299e48f (127.28.152.130:44119)
I20260926 02:14:22.311299 5101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:38831
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:14:22.347811 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36985
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:22.674294 5429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:22.674625 5429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:22.674762 5429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:22.682231 5429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:22.682384 5429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:22.682534 5429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:14:22.692607 5429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:22.694011 5429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:22.695333 5429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:22.708446 5434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:22.708479 5437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:22.708419 5435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:22.709779 5429 server_base.cc:1034] running on GCE node
I20260926 02:14:22.710480 5429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:22.711475 5429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:22.712703 5429 hybrid_clock.cc:648] HybridClock initialized: now 1790388862712647 us; error 51 us; skew 500 ppm
I20260926 02:14:22.713035 5429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:22 dist-test-slave-hllj krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388862, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:22.716167 5429 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:14:22.717723 5429 webserver.cc:460] Webserver started at http://127.28.152.131:44413/ using document root <none> and password file <none>
I20260926 02:14:22.718413 5429 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:22.718536 5429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:22.718822 5429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:22.721765 5429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "414fdcf83a224e2b93cfc35269a720d1"
format_stamp: "Formatted at 2026-09-26 02:14:22 on dist-test-slave-hllj"
server_key: "36627d646ad750122aa57aad2250b7ac"
server_key_iv: "ca8fcfc7cf7e4da09876c566c56c4955"
server_key_version: "encryptionkey@0"
I20260926 02:14:22.722604 5429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "414fdcf83a224e2b93cfc35269a720d1"
format_stamp: "Formatted at 2026-09-26 02:14:22 on dist-test-slave-hllj"
server_key: "36627d646ad750122aa57aad2250b7ac"
server_key_iv: "ca8fcfc7cf7e4da09876c566c56c4955"
server_key_version: "encryptionkey@0"
I20260926 02:14:22.728288 5429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:14:22.732458 5444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:22.734294 5429 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 02:14:22.734472 5429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "414fdcf83a224e2b93cfc35269a720d1"
format_stamp: "Formatted at 2026-09-26 02:14:22 on dist-test-slave-hllj"
server_key: "36627d646ad750122aa57aad2250b7ac"
server_key_iv: "ca8fcfc7cf7e4da09876c566c56c4955"
server_key_version: "encryptionkey@0"
I20260926 02:14:22.734660 5429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:22.749692 5429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:22.754179 5429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:22.754448 5429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:22.755411 5429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:22.756758 5429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:22.756832 5429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:22.756922 5429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:22.756978 5429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:22.787582 5429 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:38995
I20260926 02:14:22.787622 5556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:38995 every 8 connection(s)
I20260926 02:14:22.788830 5429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:14:22.789070 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 5429
I20260926 02:14:22.789265 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:14:22.789865 29282 external_mini_cluster.cc:1428] Setting key 1c48574e40fd7a38008f5087087a9d86
Sep 26 02:14:22 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388862, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:22.809863 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:22.791672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:55079 (local address 127.28.152.190:36985)
0926 02:14:22.791839 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:22.791844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:22.793394 (+ 1550us) server_negotiation.cc:407] Connection header received
0926 02:14:22.795160 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:22.795164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:22.795231 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:22.795315 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:22.796951 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:22.797811 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:22.799482 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:22.799941 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:22.800051 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:22.803622 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:22.803628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:22.803632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:22.803667 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:22.806450 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:22.807231 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:22.807236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:22.807240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:22.807377 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:22.807980 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:22.807984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:22.807986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:22.808285 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:22.808421 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:22.809276 (+ 855us) server_negotiation.cc:299] Negotiation successful
0926 02:14:22.809513 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:14:22.810900 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:22.792339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:55079)
0926 02:14:22.793218 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:22.793276 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:22.794904 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:22.795567 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:22.795576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:22.796126 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:22.796747 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:22.796767 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:22.797995 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:22.797999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:22.799315 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:22.799327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:22.800161 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:22.800167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:22.800396 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:22.801024 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:22.801047 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:22.803324 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:22.806614 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:22.806625 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:22.806644 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:22.807108 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:22.807499 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:22.807558 (+ 59us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:22.807571 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:22.807814 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:22.808425 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:22.808435 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:22.808869 (+ 434us) client_negotiation.cc:769] Sending connection context
0926 02:14:22.809157 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 02:14:22.809754 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":310,"threads_started":1}
I20260926 02:14:22.812345 5557 heartbeater.cc:347] Connected to a master server at 127.28.152.190:36985
I20260926 02:14:22.812662 5557 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:22.813189 5290 heartbeater.cc:503] Master 127.28.152.190:36985 was elected leader, sending a full tablet report...
I20260926 02:14:22.813473 5557 heartbeater.cc:511] Master 127.28.152.190:36985 requested a full tablet report, sending...
I20260926 02:14:22.814911 5101 ts_manager.cc:194] Registered new tserver with Master: 414fdcf83a224e2b93cfc35269a720d1 (127.28.152.131:38995)
I20260926 02:14:22.815670 5101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:55079
I20260926 02:14:22.818204 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab.
Entry for principal oryx/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab.
WARNING: no policy specified for oryx/127.28.152.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab.
Entry for principal oryx/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab.
WARNING: no policy specified for oryx/127.28.152.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab.
Entry for principal oryx/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab.
I20260926 02:14:22.957013 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 5162
Sep 26 02:14:22 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:14:22.974426 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:22.958212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.28.152.129:37793)
0926 02:14:22.958863 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:22.958870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:22.958899 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:14:22.958957 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:22.958963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:22.959181 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:22.959327 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:22.960823 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:22.961916 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:22.963307 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:22.963940 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:22.964048 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:22.968269 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:22.968285 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:22.968302 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:22.968343 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:22.971352 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:22.972208 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:22.972217 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:22.972220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:22.972360 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:22.972763 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:22.972768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:22.972770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:22.973173 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:22.973488 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:22.973794 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 02:14:22.974067 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":342,"threads_started":1}
I20260926 02:14:22.975785 5285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:23.058254 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 5162
I20260926 02:14:23.071336 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 5296
Sep 26 02:14:23 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:14:23.086931 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:23.072188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.28.152.130:44119)
0926 02:14:23.072687 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:23.072695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:23.072722 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:14:23.072791 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:23.072798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:23.073069 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:23.073250 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:23.074432 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:23.075512 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:23.076841 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:23.077529 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:23.077649 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:23.081688 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:23.081704 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:23.081721 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:23.081762 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:23.084391 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:23.085047 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:23.085054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:23.085056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:23.085264 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:23.085569 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:23.085571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:23.085573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:23.085907 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:23.086126 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:23.086380 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 02:14:23.086638 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":242,"threads_started":1}
I20260926 02:14:23.088258 5419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:23.169533 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 5296
I20260926 02:14:23.184624 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 5429
Sep 26 02:14:23 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:14:23.201416 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:23.185662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.28.152.131:38995)
0926 02:14:23.186402 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:23.186408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:23.186441 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:14:23.186503 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:23.186509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:23.186724 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:23.186878 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:23.188253 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:23.189456 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:23.190771 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:23.191354 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:23.191473 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:23.195867 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:23.195880 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:23.195896 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:23.195937 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:23.198691 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:23.199369 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:23.199377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:23.199379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:23.199503 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:23.199854 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:23.199858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:23.199860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:23.200208 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:23.200503 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:23.200811 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 02:14:23.201078 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":402,"threads_started":1}
I20260926 02:14:23.202700 5552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:23.284861 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 5429
I20260926 02:14:23.298025 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 5069
I20260926 02:14:23.299440 5131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:23.400463 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 5069
I20260926 02:14:23.417721 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:36985
--webserver_interface=127.28.152.190
--webserver_port=44559
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:36985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:23.742076 5574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:23.742378 5574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:23.742506 5574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:23.749782 5574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:14:23.749903 5574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:23.749951 5574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:23.750030 5574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:14:23.750083 5574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:14:23.760921 5574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:36985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:36985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=44559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:23.762301 5574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:23.763624 5574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:23.775463 5579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:23.775559 5582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:23.775547 5580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:23.776233 5574 server_base.cc:1034] running on GCE node
I20260926 02:14:23.777050 5574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:23.778730 5574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:23.779989 5574 hybrid_clock.cc:648] HybridClock initialized: now 1790388863779909 us; error 58 us; skew 500 ppm
I20260926 02:14:23.780352 5574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:23 dist-test-slave-hllj krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388863, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:23.783535 5574 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:14:23.785099 5574 webserver.cc:460] Webserver started at http://127.28.152.190:44559/ using document root <none> and password file <none>
I20260926 02:14:23.785861 5574 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:23.786005 5574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:23.791857 5574 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 02:14:23.794950 5589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:23.796455 5574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:14:23.796636 5574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "9f17858b9b874e8f81fae45e0fac9546"
format_stamp: "Formatted at 2026-09-26 02:14:21 on dist-test-slave-hllj"
server_key: "95f558f88c3f1afe832c5265ae03d5cf"
server_key_iv: "6f76cd33225d1381de302601c5b72239"
server_key_version: "encryptionkey@0"
I20260926 02:14:23.797710 5574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:23.821966 5574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:23.826287 5574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:23.826563 5574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:23.850376 5574 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:36985
I20260926 02:14:23.850389 5640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:36985 every 8 connection(s)
I20260926 02:14:23.851941 5574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:14:23.853951 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 5574
I20260926 02:14:23.854696 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:37793
--local_ip_for_outbound_sockets=127.28.152.129
--tserver_master_addrs=127.28.152.190:36985
--webserver_port=45595
--webserver_interface=127.28.152.129
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.152.129
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:14:23.859048 5641 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:14:23.862545 5641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546: Bootstrap starting.
I20260926 02:14:23.881289 5641 log.cc:824] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:23.890179 5641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:14:23.890702 5641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546: Bootstrap complete.
I20260926 02:14:23.899863 5641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } }
I20260926 02:14:23.900761 5641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f17858b9b874e8f81fae45e0fac9546, State: Initialized, Role: FOLLOWER
I20260926 02:14:23.901510 5641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } }
I20260926 02:14:23.901746 5641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:14:23.901853 5641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:14:23.902009 5641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:14:23.906080 5641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } }
I20260926 02:14:23.906623 5641 leader_election.cc:302] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f17858b9b874e8f81fae45e0fac9546; no voters:
I20260926 02:14:23.907219 5641 leader_election.cc:288] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:14:23.907361 5645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:14:23.908546 5645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [term 2 LEADER]: Becoming Leader. State: Replica: 9f17858b9b874e8f81fae45e0fac9546, State: Running, Role: LEADER
I20260926 02:14:23.909215 5645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } }
I20260926 02:14:23.909415 5641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:14:23.911777 5647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f17858b9b874e8f81fae45e0fac9546. Latest consensus state: current_term: 2 leader_uuid: "9f17858b9b874e8f81fae45e0fac9546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } } }
I20260926 02:14:23.912153 5647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:23.911885 5646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9f17858b9b874e8f81fae45e0fac9546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f17858b9b874e8f81fae45e0fac9546" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36985 } } }
I20260926 02:14:23.912278 5646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:23.922209 5652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:14:23.928673 5652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:14:23.930174 5652 catalog_manager.cc:1259] Loaded cluster ID: 184c7460bd6b4964b880624c9b91a8fb
I20260926 02:14:23.930361 5652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:14:23.934335 5652 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:14:23.935940 5652 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f17858b9b874e8f81fae45e0fac9546: Loaded TSK: 0
I20260926 02:14:23.937779 5652 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:14:24.224323 5643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:24.224653 5643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:24.224776 5643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:24.232400 5643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:24.232543 5643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:24.232685 5643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:14:24.242950 5643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.129.keytab
--principal=oryx/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:37793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=45595
--tserver_master_addrs=127.28.152.190:36985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:24.244391 5643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:24.245836 5643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:24.260617 5668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:24.260617 5667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:24.261395 5670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:24.261842 5643 server_base.cc:1034] running on GCE node
I20260926 02:14:24.262498 5643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:24.263521 5643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:24.264741 5643 hybrid_clock.cc:648] HybridClock initialized: now 1790388864264676 us; error 59 us; skew 500 ppm
I20260926 02:14:24.265118 5643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388864, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:24.268501 5643 init.cc:377] Logged in from keytab as oryx/127.28.152.129@KRBTEST.COM (short username oryx)
I20260926 02:14:24.270082 5643 webserver.cc:460] Webserver started at http://127.28.152.129:45595/ using document root <none> and password file <none>
I20260926 02:14:24.270782 5643 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:24.270913 5643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:24.276870 5643 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 02:14:24.279898 5677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:24.281754 5643 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:14:24.281930 5643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "b413609c89634a228daa6a6a5a989483"
format_stamp: "Formatted at 2026-09-26 02:14:21 on dist-test-slave-hllj"
server_key: "55e88b26bec8abda12c48acf6d2d82b5"
server_key_iv: "735f6f303d144926af2e948912407691"
server_key_version: "encryptionkey@0"
I20260926 02:14:24.282933 5643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:24.301121 5643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:24.305783 5643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:24.306176 5643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:24.307358 5643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:24.308823 5643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:24.308933 5643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:24.309041 5643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:24.309115 5643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:24.343636 5643 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:37793
I20260926 02:14:24.343703 5789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:37793 every 8 connection(s)
I20260926 02:14:24.345041 5643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:14:24.346913 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 5643
I20260926 02:14:24.347539 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:44119
--local_ip_for_outbound_sockets=127.28.152.130
--tserver_master_addrs=127.28.152.190:36985
--webserver_port=46535
--webserver_interface=127.28.152.130
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.152.130
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:24.368336 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:24.348496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:50963)
0926 02:14:24.349711 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:24.349761 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:24.351493 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:24.353212 (+ 1719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:24.353221 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:24.353835 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:24.354439 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.354457 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.356710 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.356715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.358095 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.358109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.359478 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.359483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.359710 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.363444 (+ 3734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:24.363471 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:24.367328 (+ 3857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":714,"threads_started":1}
W20260926 02:14:24.369025 5790 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:36985 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.152.190:36985: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
W20260926 02:14:24.369230 5792 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:24.348477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:50963 (local address 127.28.152.190:36985)
0926 02:14:24.349271 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:24.349288 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:24.349912 (+ 624us) server_negotiation.cc:407] Connection header received
0926 02:14:24.352127 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:24.352146 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:24.352622 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:24.353028 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:24.354659 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.356507 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.358245 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.359267 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.359450 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.367838 (+ 8388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:50963: BlockingRecv error: recv got EOF from 127.28.152.129:50963 (error 108)
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":285,"threads_started":1}
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:24.378573 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:24.369671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:50937)
0926 02:14:24.370042 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:24.370061 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:24.370177 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:24.370518 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:24.370523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:24.370953 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:24.371141 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.371152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.372167 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.372173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.373719 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.373735 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.374505 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.374511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.374692 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.375254 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:24.375276 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:24.378299 (+ 3023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233}
W20260926 02:14:24.378921 5792 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:24.370114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:50937 (local address 127.28.152.190:36985)
0926 02:14:24.370261 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:24.370265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:24.370281 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:14:24.370331 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:24.370334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:24.370389 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:24.370467 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:24.371283 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.372037 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.373858 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.374346 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.374455 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.378519 (+ 4064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:50937: BlockingRecv error: recv got EOF from 127.28.152.129:50937 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:24.386897 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:24.379260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:45713)
0926 02:14:24.379414 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:24.379429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:24.379517 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:24.379850 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:24.379855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:24.380138 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:24.380281 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.380289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.381291 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.381296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.382200 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.382211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.383023 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.383030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.383188 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.383720 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:24.383736 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:24.386585 (+ 2849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 02:14:24.387230 5792 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:24.379435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:45713 (local address 127.28.152.190:36985)
0926 02:14:24.379603 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:24.379609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:24.379624 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:14:24.379668 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:24.379671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:24.379734 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:24.379808 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:24.380414 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.381159 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.382331 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.382776 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.382878 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.386781 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:45713: BlockingRecv error: recv got EOF from 127.28.152.129:45713 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 02:14:24.387228 5790 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:14:24.682657 5794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:24.682974 5794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:24.683074 5794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:24.690529 5794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:24.690649 5794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:24.690765 5794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:14:24.700675 5794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.130.keytab
--principal=oryx/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:44119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=46535
--tserver_master_addrs=127.28.152.190:36985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:24.702099 5794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:24.703444 5794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:24.716904 5802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:24.717010 5799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:24.717264 5800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:24.717814 5794 server_base.cc:1034] running on GCE node
I20260926 02:14:24.718483 5794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:24.719564 5794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:24.720801 5794 hybrid_clock.cc:648] HybridClock initialized: now 1790388864720740 us; error 53 us; skew 500 ppm
I20260926 02:14:24.721176 5794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388864, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:24.724478 5794 init.cc:377] Logged in from keytab as oryx/127.28.152.130@KRBTEST.COM (short username oryx)
I20260926 02:14:24.726042 5794 webserver.cc:460] Webserver started at http://127.28.152.130:46535/ using document root <none> and password file <none>
I20260926 02:14:24.726774 5794 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:24.726910 5794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:24.732539 5794 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:14:24.735450 5809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:24.736953 5794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:14:24.737156 5794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "1b019d8b753640cc94168a219299e48f"
format_stamp: "Formatted at 2026-09-26 02:14:22 on dist-test-slave-hllj"
server_key: "5b7df5e6e41fdf92a4684d214d98b9d0"
server_key_iv: "6d3f389f650c9c68c91efdf67a9a20c0"
server_key_version: "encryptionkey@0"
I20260926 02:14:24.738194 5794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:24.758426 5794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:24.762775 5794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:24.763037 5794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:24.764015 5794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:24.765388 5794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:24.765481 5794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:24.765594 5794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:24.765662 5794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:24.797230 5794 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:44119
I20260926 02:14:24.797274 5921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:44119 every 8 connection(s)
I20260926 02:14:24.798673 5794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:14:24.800989 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 5794
I20260926 02:14:24.801697 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:38995
--local_ip_for_outbound_sockets=127.28.152.131
--tserver_master_addrs=127.28.152.190:36985
--webserver_port=44413
--webserver_interface=127.28.152.131
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:42275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.152.131
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:14:24.816401 5792 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:24.801251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:45927 (local address 127.28.152.190:36985)
0926 02:14:24.801435 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:24.801442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:24.803081 (+ 1639us) server_negotiation.cc:407] Connection header received
0926 02:14:24.805520 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:24.805525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:24.805596 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:24.805690 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:24.807740 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.808478 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.810310 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.810936 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.811078 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.816173 (+ 5095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:45927: BlockingRecv error: recv got EOF from 127.28.152.130:45927 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:14:24.816973 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:24.801853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:45927)
0926 02:14:24.802907 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:24.802956 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:24.805159 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:24.805973 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:24.805986 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:24.806719 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:24.807527 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.807553 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.808649 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.808655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.810161 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.810175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.811220 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.811226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.811466 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.812140 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:24.812168 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:24.815888 (+ 3720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":296,"threads_started":1}
W20260926 02:14:24.817781 5922 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:36985 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.152.190:36985: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:24.827165 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:24.818394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:51765)
0926 02:14:24.818594 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:24.818614 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:24.818733 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:24.819043 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:24.819048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:24.819470 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:24.819675 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.819688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.820669 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.820675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.822136 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.822154 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.823009 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.823016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.823259 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.823917 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:24.823939 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:24.826864 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 02:14:24.827442 5792 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:24.818533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:51765 (local address 127.28.152.190:36985)
0926 02:14:24.818722 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:24.818726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:24.818743 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:14:24.818847 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:24.818851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:24.818907 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:24.818983 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:24.819816 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.820514 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.822284 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.822803 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.822926 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.827100 (+ 4174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:51765: BlockingRecv error: recv got EOF from 127.28.152.130:51765 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:24 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:24.836984 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:24.827937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:40431)
0926 02:14:24.828122 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:24.828140 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:24.828243 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:24.828546 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:24.828551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:24.828959 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:24.829179 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.829191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.830170 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.830176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.831678 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:24.831694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.832351 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.832358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:24.832567 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.833165 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:24.833188 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:24.836680 (+ 3492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:14:24.837064 5792 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:24.828063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:40431 (local address 127.28.152.190:36985)
0926 02:14:24.828234 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:24.828238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:24.828253 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:14:24.828359 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:24.828363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:24.828430 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:24.828504 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:24.829296 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.830021 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.831827 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:24.832216 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:24.832308 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:24.836893 (+ 4585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:40431: BlockingRecv error: recv got EOF from 127.28.152.130:40431 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 02:14:24.837365 5922 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:14:25.139204 5925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:25.139505 5925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:25.139598 5925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:25.146914 5925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:25.147035 5925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:25.147157 5925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:14:25.156853 5925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:42275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/krb5kdc/oryx_127.28.152.131.keytab
--principal=oryx/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:38995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=44413
--tserver_master_addrs=127.28.152.190:36985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:25.158221 5925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:25.159516 5925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:25.172585 5933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:25.172549 5931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:25.172793 5930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:25.173420 5925 server_base.cc:1034] running on GCE node
I20260926 02:14:25.174064 5925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:25.174999 5925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:25.176214 5925 hybrid_clock.cc:648] HybridClock initialized: now 1790388865176156 us; error 49 us; skew 500 ppm
I20260926 02:14:25.176550 5925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:25 dist-test-slave-hllj krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388865, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:25.179631 5925 init.cc:377] Logged in from keytab as oryx/127.28.152.131@KRBTEST.COM (short username oryx)
I20260926 02:14:25.181242 5925 webserver.cc:460] Webserver started at http://127.28.152.131:44413/ using document root <none> and password file <none>
I20260926 02:14:25.181933 5925 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:25.182058 5925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:25.187731 5925 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:14:25.190778 5941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:25.192373 5925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:14:25.192564 5925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "414fdcf83a224e2b93cfc35269a720d1"
format_stamp: "Formatted at 2026-09-26 02:14:22 on dist-test-slave-hllj"
server_key: "36627d646ad750122aa57aad2250b7ac"
server_key_iv: "ca8fcfc7cf7e4da09876c566c56c4955"
server_key_version: "encryptionkey@0"
I20260926 02:14:25.193683 5925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:25.231336 5925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:25.235808 5925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:25.236096 5925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:25.237102 5925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:25.238493 5925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:25.238593 5925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:25.238704 5925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:25.238786 5925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:25.277730 5925 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:38995
I20260926 02:14:25.277773 6054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:38995 every 8 connection(s)
I20260926 02:14:25.279707 5925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:14:25.286924 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 5925
Sep 26 02:14:25 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:25 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:14:25.299593 5792 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:25.282836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:45955 (local address 127.28.152.190:36985)
0926 02:14:25.283063 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:25.283069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:25.284492 (+ 1423us) server_negotiation.cc:407] Connection header received
0926 02:14:25.287312 (+ 2820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:25.287318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:25.287412 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:25.287532 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:25.289858 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.290612 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.292350 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.293096 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.293244 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:25.299302 (+ 6058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:45955: BlockingRecv error: recv got EOF from 127.28.152.131:45955 (error 108)
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":84}
I20260926 02:14:25.300529 6057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:25.283404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:45955)
0926 02:14:25.284294 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:25.284345 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:25.286858 (+ 2513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:25.287873 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:25.287887 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:25.288733 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:25.289640 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:25.289669 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.290792 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.290797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:25.292196 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:25.292209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.293472 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.293477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:25.293688 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:25.294351 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:25.294375 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:25.299109 (+ 4734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":323,"threads_started":1}
W20260926 02:14:25.301549 6055 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:36985 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.152.190:36985: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Sep 26 02:14:25 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:25 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:25.317029 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:25.289982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.28.152.190:36985)
0926 02:14:25.292798 (+ 2816us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:25.292804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:25.292821 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:14:25.292867 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:25.292872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:25.292950 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:25.293057 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:25.295102 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.296031 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.297646 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.298164 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.298294 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:25.301405 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:25.301422 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:25.301439 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:25.301482 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:25.305347 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:25.306144 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:25.306154 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:25.306162 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:25.306351 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:25.307347 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:25.307351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:25.307353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:25.307912 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:25.308179 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:25.315738 (+ 7559us) server_negotiation.cc:299] Negotiation successful
0926 02:14:25.316621 (+ 883us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2685,"thread_start_us":2548,"threads_started":1}
I20260926 02:14:25.317466 6057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:25.302230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:55853)
0926 02:14:25.302506 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:25.302525 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:25.302635 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:25.302971 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:25.302976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:25.303416 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:25.303604 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:25.303616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.305029 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.305035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:25.306577 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:25.306594 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.307281 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.307287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:25.307468 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:25.309185 (+ 1717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:25.309208 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:25.317206 (+ 7998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":20}
W20260926 02:14:25.317682 5792 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:25.302462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:55853 (local address 127.28.152.190:36985)
0926 02:14:25.302658 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:25.302664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:25.302687 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:14:25.302739 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:25.302747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:25.302820 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:25.302938 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:25.303761 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.304853 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.306727 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.307139 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.307230 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:25.317423 (+ 10193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:55853: BlockingRecv error: recv got EOF from 127.28.152.131:55853 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 02:14:25 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:25 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:25.328166 6057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:25.318233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:44151)
0926 02:14:25.318417 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:25.318437 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:25.318550 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:25.320123 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:25.320129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:25.320429 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:25.320655 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:25.320668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.322113 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.322118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:25.323122 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:25.323136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.324074 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.324079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:25.324239 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:25.324837 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:25.324854 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:25.327846 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 02:14:25.328294 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:25.318310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:44151 (local address 127.28.152.190:36985)
0926 02:14:25.319775 (+ 1465us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:25.319781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:25.319802 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:14:25.319856 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:25.319861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:25.319945 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:25.320038 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:25.320804 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.321951 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.323258 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.323914 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.324052 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:25.328040 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:44151: BlockingRecv error: recv got EOF from 127.28.152.131:44151 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260926 02:14:25.328490 6055 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 02:14:25 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:25 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:25.396322 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:25.387830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:43123)
0926 02:14:25.388314 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:25.388329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:25.388489 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:25.388785 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:25.388788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:25.389242 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:25.389497 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:25.389508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.390601 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.390605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:25.391608 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:25.391620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.392371 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.392376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:25.392525 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:25.393148 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:25.393167 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:25.396001 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":233,"threads_started":1}
W20260926 02:14:25.396453 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:25.387944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:43123 (local address 127.28.152.190:36985)
0926 02:14:25.388192 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:25.388199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:25.388401 (+ 202us) server_negotiation.cc:407] Connection header received
0926 02:14:25.388610 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:25.388615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:25.388681 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:25.388754 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:25.389674 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.390431 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.391752 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.392179 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.392277 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:25.396229 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:43123: BlockingRecv error: recv got EOF from 127.28.152.129:43123 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 02:14:25 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:25 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:25.847328 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:25.838041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:46031)
0926 02:14:25.838549 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:25.838565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:25.838700 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:25.839028 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:25.839033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:25.839487 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:25.839675 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:25.839684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.840707 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.840713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:25.841929 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:25.841944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.843027 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.843034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:25.843212 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:25.843834 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:25.843852 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:25.847000 (+ 3148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":264,"threads_started":1}
W20260926 02:14:25.847699 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:25.838162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:46031 (local address 127.28.152.190:36985)
0926 02:14:25.838375 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:25.838381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:25.838649 (+ 268us) server_negotiation.cc:407] Connection header received
0926 02:14:25.838827 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:25.838830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:25.838896 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:25.838982 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:25.839815 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.840571 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.842094 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:25.842804 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:25.842936 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:25.847321 (+ 4385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:46031: BlockingRecv error: recv got EOF from 127.28.152.130:46031 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 02:14:26 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:26 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:26.337801 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:26.329108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:53691)
0926 02:14:26.329671 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:26.329686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:26.329804 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:26.330166 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:26.330169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:26.330650 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:26.330843 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:26.330852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.331978 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.331984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:26.333017 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:26.333030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.333937 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.333942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:26.334073 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:26.334669 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:26.334686 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:26.337503 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":265,"threads_started":1}
W20260926 02:14:26.338038 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:26.329291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:53691 (local address 127.28.152.190:36985)
0926 02:14:26.329478 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:26.329484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:26.329772 (+ 288us) server_negotiation.cc:407] Connection header received
0926 02:14:26.329904 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:26.329910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:26.330014 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:26.330133 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:26.331025 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.331806 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.333232 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.333727 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.333839 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:26.337770 (+ 3931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:53691: BlockingRecv error: recv got EOF from 127.28.152.131:53691 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 02:14:26 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:26 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:26.405936 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:26.397509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:54651)
0926 02:14:26.398031 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:26.398051 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:26.398210 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:26.398609 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:26.398622 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:26.399083 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:26.399302 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:26.399313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.400533 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.400536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:26.401537 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:26.401551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.402313 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.402318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:26.402427 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:26.402974 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:26.402992 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:26.405696 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":279,"threads_started":1}
W20260926 02:14:26.406098 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:26.397585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:54651 (local address 127.28.152.190:36985)
0926 02:14:26.397806 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:26.397813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:26.398161 (+ 348us) server_negotiation.cc:407] Connection header received
0926 02:14:26.398367 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:26.398372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:26.398453 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:26.398550 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:26.399446 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.400408 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.401665 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.402168 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.402284 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:26.405897 (+ 3613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:54651: BlockingRecv error: recv got EOF from 127.28.152.129:54651 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 02:14:26 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:26 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:26.859033 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:26.848528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:45983)
0926 02:14:26.849496 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:26.849517 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:26.849823 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:26.850352 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:26.850357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:26.850831 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:26.851065 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:26.851077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.852540 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.852546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:26.854025 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:26.854040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.854811 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.854818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:26.855013 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:26.855606 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:26.855628 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:26.858709 (+ 3081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":615,"threads_started":1}
W20260926 02:14:26.859153 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:26.848683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:45983 (local address 127.28.152.190:36985)
0926 02:14:26.848869 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:26.848875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:26.849598 (+ 723us) server_negotiation.cc:407] Connection header received
0926 02:14:26.849982 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:26.849986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:26.850058 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:26.850121 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:26.851229 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.852370 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.854175 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:26.854648 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:26.854750 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:26.858927 (+ 4177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:45983: BlockingRecv error: recv got EOF from 127.28.152.130:45983 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 02:14:27 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:27 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:27.347254 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:27.338760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:39335)
0926 02:14:27.339417 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:27.339433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:27.339601 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:27.339926 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:27.339930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:27.340321 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:27.340514 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:27.340524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.341540 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.341544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:27.342530 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:27.342542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.343290 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.343296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:27.343481 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:27.344091 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:27.344109 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:27.346918 (+ 2809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":428,"threads_started":1}
W20260926 02:14:27.347393 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:27.338944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:39335 (local address 127.28.152.190:36985)
0926 02:14:27.339130 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:27.339136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:27.339535 (+ 399us) server_negotiation.cc:407] Connection header received
0926 02:14:27.339730 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:27.339735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:27.339798 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:27.339873 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:27.340652 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.341406 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.342667 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.343146 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.343252 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:27.347169 (+ 3917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:39335: BlockingRecv error: recv got EOF from 127.28.152.131:39335 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 02:14:27 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:27 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:14:27.415978 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:27.407084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:51245 (local address 127.28.152.190:36985)
0926 02:14:27.407256 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:27.407262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:27.407643 (+ 381us) server_negotiation.cc:407] Connection header received
0926 02:14:27.407806 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:27.407810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:27.407873 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:27.407956 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:27.408715 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.409533 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.410814 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.411366 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.411490 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:27.415792 (+ 4302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:51245: BlockingRecv error: recv got EOF from 127.28.152.129:51245 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 02:14:27.415971 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:27.407034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:51245)
0926 02:14:27.407530 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:27.407546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:27.407685 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:27.408000 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:27.408006 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:27.408372 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:27.408536 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:27.408545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.409681 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.409685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:27.410675 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:27.410687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.411521 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.411527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:27.411703 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:27.412378 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:27.412402 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:27.415606 (+ 3204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":265,"threads_started":1}
Sep 26 02:14:27 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:27 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:27.869081 6067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:27.860151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:43111)
0926 02:14:27.860898 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:27.860917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:27.861059 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:27.861397 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:27.861402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:27.861842 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:27.862062 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:27.862076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.863128 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.863134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:27.864043 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:27.864054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.864778 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.864784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:27.864967 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:27.865615 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:27.865633 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:27.868727 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":497,"threads_started":1}
W20260926 02:14:27.869246 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:27.860261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:43111 (local address 127.28.152.190:36985)
0926 02:14:27.860698 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:27.860706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:27.861038 (+ 332us) server_negotiation.cc:407] Connection header received
0926 02:14:27.861184 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:27.861190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:27.861258 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:27.861343 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:27.862188 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.862962 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.864195 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:27.864633 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:27.864745 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:27.868976 (+ 4231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:43111: BlockingRecv error: recv got EOF from 127.28.152.130:43111 (error 108)
Metrics: {"server-negotiator.queue_time_us":292}
Sep 26 02:14:28 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:28 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:28.356801 6068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:28.348439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:56953)
0926 02:14:28.349010 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:28.349030 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:28.349191 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:28.349513 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:28.349517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:28.349956 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:28.350137 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:28.350148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.351162 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.351166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:28.352064 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:28.352076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.352782 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.352787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:28.352927 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:28.353622 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:28.353651 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:28.356539 (+ 2888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":383,"threads_started":1}
W20260926 02:14:28.357033 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:28.348592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:56953 (local address 127.28.152.190:36985)
0926 02:14:28.348798 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:28.348805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:28.349124 (+ 319us) server_negotiation.cc:407] Connection header received
0926 02:14:28.349324 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:28.349329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:28.349389 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:28.349466 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:28.350277 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.351006 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.352205 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.352627 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.352725 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:28.356781 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:56953: BlockingRecv error: recv got EOF from 127.28.152.131:56953 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 02:14:28 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:28 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:28.425472 6069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:28.416911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:58569)
0926 02:14:28.417409 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:28.417425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:28.417544 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:28.417911 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:28.417915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:28.418263 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:28.418415 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:28.418424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.419536 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.419541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:28.420520 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:28.420534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.421232 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.421237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:28.421363 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:28.421973 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:28.421992 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:28.425165 (+ 3173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":282,"threads_started":1}
W20260926 02:14:28.425601 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:28.417192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:58569 (local address 127.28.152.190:36985)
0926 02:14:28.417355 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:28.417359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:28.417494 (+ 135us) server_negotiation.cc:407] Connection header received
0926 02:14:28.417637 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:28.417641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:28.417700 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:28.417790 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:28.418633 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.419364 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.420671 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.421057 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.421186 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:28.425389 (+ 4203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:58569: BlockingRecv error: recv got EOF from 127.28.152.129:58569 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 02:14:28 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:28 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:28.878588 6070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:28.870326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:33485)
0926 02:14:28.870826 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:28.870848 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:28.871022 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:28.871352 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:28.871357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:28.871720 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:28.871898 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:28.871911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.873073 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.873076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:28.874110 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:28.874121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.874820 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.874824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:28.874951 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:28.875521 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:28.875540 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:28.878361 (+ 2821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":267,"threads_started":1}
W20260926 02:14:28.878866 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:28.870462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:33485 (local address 127.28.152.190:36985)
0926 02:14:28.870681 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:28.870686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:28.870945 (+ 259us) server_negotiation.cc:407] Connection header received
0926 02:14:28.871167 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:28.871171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:28.871230 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:28.871307 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:28.872054 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.872960 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.874230 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:28.874664 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:28.874763 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:28.878609 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:33485: BlockingRecv error: recv got EOF from 127.28.152.130:33485 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 02:14:29 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:29 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:29.371834 6071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:29.357880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:35041)
0926 02:14:29.358369 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:29.358391 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:29.358540 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:29.359176 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:29.359181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:29.359592 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:29.359798 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:29.359810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.361171 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.361177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:29.362202 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:29.362216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.362956 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.362961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:29.363103 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:29.368331 (+ 5228us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:29.368355 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:29.371545 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":241,"threads_started":1}
W20260926 02:14:29.372160 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:29.358081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:35041 (local address 127.28.152.190:36985)
0926 02:14:29.358295 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:29.358301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:29.358477 (+ 176us) server_negotiation.cc:407] Connection header received
0926 02:14:29.358685 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:29.358691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:29.358774 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:29.358886 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:29.359984 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.360983 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.362356 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.362799 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.362898 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:29.371872 (+ 8974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:35041: BlockingRecv error: recv got EOF from 127.28.152.131:35041 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 02:14:29 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:29 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:29.435320 6072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:29.426631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:49789)
0926 02:14:29.427171 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:29.427187 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:29.427332 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:29.427739 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:29.427745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:29.428169 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:29.428337 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:29.428346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.429425 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.429429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:29.430373 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:29.430387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.431123 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.431130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:29.431313 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:29.431978 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:29.431996 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:29.435007 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":263,"threads_started":1}
W20260926 02:14:29.435796 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:29.426678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:49789 (local address 127.28.152.190:36985)
0926 02:14:29.426879 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:29.426887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:29.427282 (+ 395us) server_negotiation.cc:407] Connection header received
0926 02:14:29.427447 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:29.427453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:29.427542 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:29.427626 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:29.428486 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.429289 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.430524 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.430966 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.431078 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:29.435314 (+ 4236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:49789: BlockingRecv error: recv got EOF from 127.28.152.129:49789 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 02:14:29 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:29 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:29.888159 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:29.879652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:43185)
0926 02:14:29.880187 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:29.880204 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:29.880325 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:29.880697 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:29.880701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:29.881093 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:29.881317 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:29.881326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.882407 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.882411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:29.883334 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:29.883346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.884058 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.884062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:29.884194 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:29.884788 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:29.884806 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:29.887916 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":238,"threads_started":1}
W20260926 02:14:29.888345 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:29.879902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:43185 (local address 127.28.152.190:36985)
0926 02:14:29.880094 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:29.880100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:29.880311 (+ 211us) server_negotiation.cc:407] Connection header received
0926 02:14:29.880417 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:29.880421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:29.880564 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:29.880663 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:29.881504 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.882292 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.883475 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:29.883946 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:29.884043 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:29.888134 (+ 4091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:43185: BlockingRecv error: recv got EOF from 127.28.152.130:43185 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 02:14:30 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:30 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:30.380650 6074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:30.372912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:56527)
0926 02:14:30.373404 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:30.373421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:30.373534 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:30.373830 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:30.373834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:30.374215 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:30.374362 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:30.374370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.375310 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.375313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:30.376236 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:30.376248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.376992 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.376996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:30.377119 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:30.377734 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:30.377752 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:30.380398 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":285,"threads_started":1}
W20260926 02:14:30.380868 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:30.373013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:56527 (local address 127.28.152.190:36985)
0926 02:14:30.373215 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:30.373219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:30.373484 (+ 265us) server_negotiation.cc:407] Connection header received
0926 02:14:30.373614 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:30.373618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:30.373712 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:30.373794 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:30.374493 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.375208 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.376377 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.376870 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.376968 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:30.380603 (+ 3635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:56527: BlockingRecv error: recv got EOF from 127.28.152.131:56527 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 02:14:30 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:30 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:30.445801 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:30.436313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:35983)
0926 02:14:30.436843 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:30.436859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:30.436979 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:30.437294 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:30.437298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:30.437696 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:30.437852 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:30.437860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.438894 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.438898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:30.440131 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:30.440151 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.441384 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.441392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:30.441580 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:30.442257 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:30.442275 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:30.445490 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":248,"threads_started":1}
W20260926 02:14:30.446164 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:30.436434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:35983 (local address 127.28.152.190:36985)
0926 02:14:30.436668 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:30.436674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:30.436942 (+ 268us) server_negotiation.cc:407] Connection header received
0926 02:14:30.437087 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:30.437091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:30.437178 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:30.437262 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:30.437995 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.438729 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.440295 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.440984 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.441117 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:30.445743 (+ 4626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:35983: BlockingRecv error: recv got EOF from 127.28.152.129:35983 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 02:14:30 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:30 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:30.897673 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:30.889067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:36189)
0926 02:14:30.889601 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:30.889621 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:30.889753 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:30.890027 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:30.890031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:30.890402 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:30.890560 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:30.890571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.891694 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.891699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:30.892731 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:30.892746 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.893605 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.893611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:30.893782 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:30.894449 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:30.894466 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:30.897391 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":298,"threads_started":1}
W20260926 02:14:30.898021 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:30.889197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:36189 (local address 127.28.152.190:36985)
0926 02:14:30.889368 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:30.889372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:30.889706 (+ 334us) server_negotiation.cc:407] Connection header received
0926 02:14:30.889842 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:30.889846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:30.889909 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:30.889994 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:30.890721 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.891462 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.892909 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:30.893444 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:30.893552 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:30.897748 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:36189: BlockingRecv error: recv got EOF from 127.28.152.130:36189 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 02:14:31 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:31 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:31.389854 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:31.381634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:58953)
0926 02:14:31.382063 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:31.382079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:31.382198 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:31.382501 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:31.382504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:31.382867 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:31.383015 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:31.383024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.383986 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.383989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:31.384915 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:31.384927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.385750 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.385757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:31.385904 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:31.386432 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:31.386448 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:31.389553 (+ 3105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":229,"threads_started":1}
W20260926 02:14:31.389953 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:31.381764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:58953 (local address 127.28.152.190:36985)
0926 02:14:31.381968 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:31.381973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:31.382163 (+ 190us) server_negotiation.cc:407] Connection header received
0926 02:14:31.382313 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:31.382316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:31.382378 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:31.382457 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:31.383132 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.383883 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.385060 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.385555 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.385669 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:31.389748 (+ 4079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:58953: BlockingRecv error: recv got EOF from 127.28.152.131:58953 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 02:14:31 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:31 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:31.455500 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:31.446790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:52095)
0926 02:14:31.447324 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:31.447342 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:31.447483 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:31.447887 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:31.447891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:31.448296 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:31.448466 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:31.448476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.449629 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.449634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:31.450741 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:31.450757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.451574 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.451580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:31.451736 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:31.452312 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:31.452330 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:31.455239 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":232,"threads_started":1}
W20260926 02:14:31.455864 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:31.446959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:52095 (local address 127.28.152.190:36985)
0926 02:14:31.447191 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:31.447197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:31.447469 (+ 272us) server_negotiation.cc:407] Connection header received
0926 02:14:31.447634 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:31.447638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:31.447712 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:31.447804 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:31.448642 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.449468 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.450882 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.451406 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.451520 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:31.455511 (+ 3991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:52095: BlockingRecv error: recv got EOF from 127.28.152.129:52095 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 02:14:31 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:31 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:31.907670 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:31.898657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:57327)
0926 02:14:31.899204 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:31.899222 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:31.899354 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:31.899835 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:31.899839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:31.900357 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:31.900543 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:31.900557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.901890 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.901896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:31.903066 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:31.903081 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.903812 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.903817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:31.903937 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:31.904476 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:31.904493 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:31.907417 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":241,"threads_started":1}
W20260926 02:14:31.907894 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:31.898760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:57327 (local address 127.28.152.190:36985)
0926 02:14:31.898932 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:31.898936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:31.899304 (+ 368us) server_negotiation.cc:407] Connection header received
0926 02:14:31.899511 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:31.899515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:31.899635 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:31.899825 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:31.900715 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.901758 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.903209 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:31.903622 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:31.903761 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:31.907649 (+ 3888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:57327: BlockingRecv error: recv got EOF from 127.28.152.130:57327 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 02:14:32 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:32 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:32.399500 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:32.390922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:44061)
0926 02:14:32.391438 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:32.391453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:32.391572 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:32.391861 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:32.391864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:32.392236 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:32.392406 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:32.392414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.393467 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.393471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:32.394394 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:32.394407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.395141 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.395146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:32.395264 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:32.395894 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:32.395922 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:32.399166 (+ 3244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":236,"threads_started":1}
W20260926 02:14:32.399629 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:32.390982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:44061 (local address 127.28.152.190:36985)
0926 02:14:32.391255 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:32.391262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:32.391543 (+ 281us) server_negotiation.cc:407] Connection header received
0926 02:14:32.391665 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:32.391670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:32.391737 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:32.391826 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:32.392520 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.393326 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.394549 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.394997 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.395103 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:32.399395 (+ 4292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:44061: BlockingRecv error: recv got EOF from 127.28.152.131:44061 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 02:14:32 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:32 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:32.464938 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:32.456627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:57707)
0926 02:14:32.457269 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:32.457287 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:32.457413 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:32.457743 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:32.457747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:32.458111 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:32.458266 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:32.458276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.459364 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.459370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:32.460304 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:32.460318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.461114 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.461122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:32.461304 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:32.461899 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:32.461918 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:32.464560 (+ 2642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":309,"threads_started":1}
W20260926 02:14:32.465036 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:32.456897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:57707 (local address 127.28.152.190:36985)
0926 02:14:32.457152 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:32.457166 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:32.457365 (+ 199us) server_negotiation.cc:407] Connection header received
0926 02:14:32.457516 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:32.457520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:32.457586 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:32.457677 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:32.458408 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.459193 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.460458 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.460923 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.461033 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:32.464809 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:57707: BlockingRecv error: recv got EOF from 127.28.152.129:57707 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 02:14:32 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:32 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:32.916997 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:32.908640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:42953)
0926 02:14:32.909190 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:32.909209 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:32.909342 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:32.909656 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:32.909660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:32.910059 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:32.910218 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:32.910227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.911470 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.911474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:32.912374 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:32.912386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.913105 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.913110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:32.913272 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:32.913804 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:32.913820 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:32.916672 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
W20260926 02:14:32.917217 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:32.908805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:42953 (local address 127.28.152.190:36985)
0926 02:14:32.908987 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:32.908991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:32.909283 (+ 292us) server_negotiation.cc:407] Connection header received
0926 02:14:32.909463 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:32.909467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:32.909530 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:32.909617 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:32.910345 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.911325 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.912506 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:32.912940 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:32.913036 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:32.916930 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:42953: BlockingRecv error: recv got EOF from 127.28.152.130:42953 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 02:14:33 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:33 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:33.408869 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:33.400427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:53011)
0926 02:14:33.400951 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:33.400967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:33.401098 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:33.401493 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:33.401497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:33.401898 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:33.402056 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:33.402085 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.403062 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.403067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:33.404021 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:33.404033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.404840 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.404844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:33.404967 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:33.405573 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:33.405592 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:33.408533 (+ 2941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":261,"threads_started":1}
W20260926 02:14:33.409014 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:33.400553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:53011 (local address 127.28.152.190:36985)
0926 02:14:33.400853 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:33.400859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:33.401048 (+ 189us) server_negotiation.cc:407] Connection header received
0926 02:14:33.401265 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:33.401269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:33.401356 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:33.401454 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:33.402212 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.402914 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.404194 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.404662 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.404768 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:33.408788 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:53011: BlockingRecv error: recv got EOF from 127.28.152.131:53011 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
Sep 26 02:14:33 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:33 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:33.475203 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:33.466131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:60787)
0926 02:14:33.466695 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:33.466712 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:33.466844 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:33.467175 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:33.467178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:33.467556 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:33.467722 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:33.467731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.468780 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.468784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:33.469896 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:33.469910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.470826 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.470832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:33.470995 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:33.471740 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:33.471765 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:33.474844 (+ 3079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":277,"threads_started":1}
W20260926 02:14:33.475379 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:33.466310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:60787 (local address 127.28.152.190:36985)
0926 02:14:33.466475 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:33.466479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:33.466829 (+ 350us) server_negotiation.cc:407] Connection header received
0926 02:14:33.466950 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:33.466953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:33.467019 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:33.467130 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:33.467874 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.468634 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.470056 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.470635 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.470766 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:33.475117 (+ 4351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:60787: BlockingRecv error: recv got EOF from 127.28.152.129:60787 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:14:33 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:33 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:33.926505 6085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:33.918074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:60371)
0926 02:14:33.918723 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:33.918740 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:33.918857 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:33.919165 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:33.919170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:33.919588 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:33.919759 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:33.919775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.920767 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.920773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:33.921758 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:33.921773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.922590 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.922598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:33.922753 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:33.923360 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:33.923379 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:33.926255 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":273,"threads_started":1}
W20260926 02:14:33.926743 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:33.918189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:60371 (local address 127.28.152.190:36985)
0926 02:14:33.918386 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:33.918392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:33.918832 (+ 440us) server_negotiation.cc:407] Connection header received
0926 02:14:33.918959 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:33.918962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:33.919024 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:33.919113 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:33.919912 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.920631 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.921909 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:33.922371 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:33.922494 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:33.926494 (+ 4000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:60371: BlockingRecv error: recv got EOF from 127.28.152.130:60371 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 02:14:34 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:34 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:34.425004 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:34.409910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:47875)
0926 02:14:34.410404 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:34.410420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:34.410540 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:34.410885 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:34.410888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:34.411283 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:34.411432 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:34.411441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.412422 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.412426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:34.413502 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:34.413521 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.414378 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.414384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:34.414534 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:34.421586 (+ 7052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:34.421613 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:34.424756 (+ 3143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":260,"threads_started":1}
W20260926 02:14:34.425215 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:34.410020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:47875 (local address 127.28.152.190:36985)
0926 02:14:34.410205 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:34.410209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:34.410497 (+ 288us) server_negotiation.cc:407] Connection header received
0926 02:14:34.410639 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:34.410643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:34.410726 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:34.410837 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:34.411556 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.412265 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.413672 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.414207 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.414307 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:34.424953 (+ 10646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:47875: BlockingRecv error: recv got EOF from 127.28.152.131:47875 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 02:14:34 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:34 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:34.484627 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:34.476231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:42349)
0926 02:14:34.476740 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:34.476760 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:34.476890 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:34.477236 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:34.477241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:34.477610 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:34.477759 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:34.477777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.478722 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.478725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:34.479651 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:34.479662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.480400 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.480406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:34.480581 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:34.481238 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:34.481260 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:34.484407 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":422,"threads_started":1}
W20260926 02:14:34.484822 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:34.476327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:42349 (local address 127.28.152.190:36985)
0926 02:14:34.476503 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:34.476507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:34.476850 (+ 343us) server_negotiation.cc:407] Connection header received
0926 02:14:34.476997 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:34.477002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:34.477073 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:34.477216 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:34.477885 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.478613 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.479796 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.480227 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.480331 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:34.484615 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:42349: BlockingRecv error: recv got EOF from 127.28.152.129:42349 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 02:14:34 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:34 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:34.935829 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:34.927502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:51143)
0926 02:14:34.928052 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:34.928068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:34.928188 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:34.928518 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:34.928522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:34.928896 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:34.929054 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:34.929062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.930129 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.930133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:34.931084 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:34.931097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.931835 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.931840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:34.931961 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:34.932561 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:34.932578 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:34.935594 (+ 3016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":334,"threads_started":1}
W20260926 02:14:34.936075 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:34.927600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:51143 (local address 127.28.152.190:36985)
0926 02:14:34.927766 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:34.927770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:34.928142 (+ 372us) server_negotiation.cc:407] Connection header received
0926 02:14:34.928286 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:34.928290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:34.928350 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:34.928461 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:34.929247 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.930001 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.931229 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:34.931647 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:34.931742 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:34.935830 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:51143: BlockingRecv error: recv got EOF from 127.28.152.130:51143 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 02:14:35 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:35 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:35.433671 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:35.425953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:53289)
0926 02:14:35.426418 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:35.426434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:35.426542 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:35.426845 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:35.426848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:35.427176 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:35.427321 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:35.427329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.428281 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.428284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:35.429238 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:35.429251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.429954 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.429958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:35.430065 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:35.430604 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:35.430621 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:35.433370 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":249,"threads_started":1}
W20260926 02:14:35.433867 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:35.426132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:53289 (local address 127.28.152.190:36985)
0926 02:14:35.426409 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:35.426415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:35.426501 (+ 86us) server_negotiation.cc:407] Connection header received
0926 02:14:35.426642 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:35.426645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:35.426720 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:35.426830 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:35.427447 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.428177 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.429370 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.429823 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.429917 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:35.433612 (+ 3695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:53289: BlockingRecv error: recv got EOF from 127.28.152.131:53289 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
Sep 26 02:14:35 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:35 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:35.493624 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:35.485472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:39403)
0926 02:14:35.485983 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:35.485999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:35.486111 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:35.486383 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:35.486388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:35.486800 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:35.487001 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:35.487014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.488100 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.488104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:35.489014 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:35.489026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.489806 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.489810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:35.489937 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:35.490522 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:35.490539 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:35.493295 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":270,"threads_started":1}
W20260926 02:14:35.493759 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:35.485561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:39403 (local address 127.28.152.190:36985)
0926 02:14:35.485738 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:35.485742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:35.486062 (+ 320us) server_negotiation.cc:407] Connection header received
0926 02:14:35.486195 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:35.486200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:35.486260 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:35.486349 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:35.487168 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.487972 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.489201 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.489665 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.489778 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:35.493531 (+ 3753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:39403: BlockingRecv error: recv got EOF from 127.28.152.129:39403 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 02:14:35 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:35 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:35.946031 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:35.936971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:60233)
0926 02:14:35.937523 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:35.937540 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:35.937666 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:35.938013 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:35.938017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:35.938471 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:35.938679 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:35.938691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.939763 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.939767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:35.940763 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:35.940777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.941704 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.941710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:35.941857 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:35.942550 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:35.942569 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:35.945794 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":313,"threads_started":1}
W20260926 02:14:35.946223 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:35.937202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:60233 (local address 127.28.152.190:36985)
0926 02:14:35.937382 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:35.937386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:35.937642 (+ 256us) server_negotiation.cc:407] Connection header received
0926 02:14:35.937777 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:35.937781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:35.937849 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:35.937929 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:35.938858 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.939606 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.940931 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:35.941465 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:35.941572 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:35.945973 (+ 4401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:60233: BlockingRecv error: recv got EOF from 127.28.152.130:60233 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 02:14:36 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:36 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:36.443568 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:36.434887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:44627)
0926 02:14:36.435412 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:36.435429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:36.435577 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:36.435922 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:36.435927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:36.436338 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:36.436512 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:36.436522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.437681 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.437686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:36.438601 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:36.438656 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.439505 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.439511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:36.439688 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:36.440289 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:36.440312 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:36.443337 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":307,"threads_started":1}
W20260926 02:14:36.443804 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:36.434984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:44627 (local address 127.28.152.190:36985)
0926 02:14:36.435251 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:36.435258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:36.435528 (+ 270us) server_negotiation.cc:407] Connection header received
0926 02:14:36.435714 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:36.435719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:36.435781 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:36.435867 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:36.436691 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.437479 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.438890 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.439343 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.439464 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:36.443537 (+ 4073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:44627: BlockingRecv error: recv got EOF from 127.28.152.131:44627 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 02:14:36 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:36 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:36.502611 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:36.494573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:40085)
0926 02:14:36.495010 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:36.495026 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:36.495141 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:36.495432 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:36.495435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:36.495847 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:36.496000 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:36.496009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.497040 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.497044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:36.498110 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:36.498124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.498854 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.498858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:36.498986 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:36.499545 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:36.499561 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:36.502293 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":248,"threads_started":1}
W20260926 02:14:36.502876 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:36.494671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:40085 (local address 127.28.152.190:36985)
0926 02:14:36.494931 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:36.494937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:36.495099 (+ 162us) server_negotiation.cc:407] Connection header received
0926 02:14:36.495250 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:36.495254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:36.495315 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:36.495386 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:36.496134 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.496909 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.498262 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.498687 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.498833 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:36.502559 (+ 3726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:40085: BlockingRecv error: recv got EOF from 127.28.152.129:40085 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Sep 26 02:14:36 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:36 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:36.955631 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:36.947012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:37507)
0926 02:14:36.947490 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:36.947510 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:36.947633 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:36.947974 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:36.947977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:36.948377 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:36.948532 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:36.948542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.949691 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.949697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:36.950669 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:36.950683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.951380 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.951385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:36.951524 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:36.952196 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:36.952221 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:36.955369 (+ 3148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":287,"threads_started":1}
W20260926 02:14:36.955932 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:36.947115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:37507 (local address 127.28.152.190:36985)
0926 02:14:36.947337 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:36.947343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:36.947594 (+ 251us) server_negotiation.cc:407] Connection header received
0926 02:14:36.947746 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:36.947750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:36.947844 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:36.947930 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:36.948677 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.949529 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.950799 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:36.951231 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:36.951328 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:36.955592 (+ 4264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:37507: BlockingRecv error: recv got EOF from 127.28.152.130:37507 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 02:14:37 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:37 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:37.452841 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:37.444614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:40363)
0926 02:14:37.445169 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:37.445186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:37.445307 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:37.445650 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:37.445654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:37.446090 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:37.446236 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:37.446248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.447291 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.447295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:37.448214 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:37.448226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.448936 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.448942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:37.449107 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:37.449774 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:37.449792 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:37.452615 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":325,"threads_started":1}
W20260926 02:14:37.453045 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:37.444785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:40363 (local address 127.28.152.190:36985)
0926 02:14:37.444952 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:37.444956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:37.445278 (+ 322us) server_negotiation.cc:407] Connection header received
0926 02:14:37.445417 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:37.445422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:37.445502 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:37.445631 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:37.446402 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.447137 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.448357 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.448791 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.448899 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:37.452822 (+ 3923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:40363: BlockingRecv error: recv got EOF from 127.28.152.131:40363 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:14:37 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:37 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:37.511732 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:37.503631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:41567)
0926 02:14:37.504190 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:37.504206 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:37.504328 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:37.504618 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:37.504622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:37.504995 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:37.505192 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:37.505204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.506228 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.506250 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:37.507328 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:37.507340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.508090 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.508097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:37.508248 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:37.508753 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:37.508770 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:37.511417 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
W20260926 02:14:37.512117 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:37.503705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:41567 (local address 127.28.152.190:36985)
0926 02:14:37.503890 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:37.503895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:37.504277 (+ 382us) server_negotiation.cc:407] Connection header received
0926 02:14:37.504418 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:37.504421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:37.504481 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:37.504567 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:37.505319 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.506084 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.507466 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.507917 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.508023 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:37.511651 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:41567: BlockingRecv error: recv got EOF from 127.28.152.129:41567 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 02:14:37 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:37 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:37.965561 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:37.956629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:34399)
0926 02:14:37.957166 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:37.957184 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:37.957324 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:37.957640 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:37.957644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:37.958061 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:37.958222 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:37.958231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.959264 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.959268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:37.960218 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:37.960230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.960966 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.960973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:37.961122 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:37.961737 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:37.961804 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:37.965230 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":244,"threads_started":1}
W20260926 02:14:37.965713 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:37.956905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:34399 (local address 127.28.152.190:36985)
0926 02:14:37.957125 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:37.957158 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:37.957285 (+ 127us) server_negotiation.cc:407] Connection header received
0926 02:14:37.957449 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:37.957453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:37.957512 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:37.957591 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:37.958375 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.959120 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.960357 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:37.960790 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:37.960899 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:37.965517 (+ 4618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:34399: BlockingRecv error: recv got EOF from 127.28.152.130:34399 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 02:14:38 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:38 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:38.462373 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:38.453777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:50393)
0926 02:14:38.454249 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:38.454266 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:38.454391 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:38.454682 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:38.454685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:38.455121 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:38.455267 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:38.455275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.456326 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.456331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:38.457377 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:38.457389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.458073 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.458079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:38.458206 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:38.458809 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:38.458828 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:38.462043 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":233,"threads_started":1}
W20260926 02:14:38.462491 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:38.453949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:50393 (local address 127.28.152.190:36985)
0926 02:14:38.454183 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:38.454191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:38.454360 (+ 169us) server_negotiation.cc:407] Connection header received
0926 02:14:38.454484 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:38.454487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:38.454551 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:38.454632 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:38.455426 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.456167 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.457495 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.457927 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.458019 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:38.462235 (+ 4216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:50393: BlockingRecv error: recv got EOF from 127.28.152.131:50393 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 02:14:38 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:38 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:38.520851 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:38.512624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:47317)
0926 02:14:38.513272 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:38.513289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:38.513403 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:38.513717 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:38.513721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:38.514072 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:38.514250 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:38.514260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.515302 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.515306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:38.516251 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:38.516264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.516955 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.516959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:38.517077 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:38.517717 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:38.517735 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:38.520528 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":316,"threads_started":1}
W20260926 02:14:38.520939 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:38.512731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:47317 (local address 127.28.152.190:36985)
0926 02:14:38.512986 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:38.512991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:38.513362 (+ 371us) server_negotiation.cc:407] Connection header received
0926 02:14:38.513491 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:38.513494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:38.513556 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:38.513638 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:38.514417 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.515160 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.516393 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.516839 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.516939 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:38.520747 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:47317: BlockingRecv error: recv got EOF from 127.28.152.129:47317 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Sep 26 02:14:38 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:38 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:38.975780 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:38.966605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:49467)
0926 02:14:38.967087 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:38.967107 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:38.967298 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:38.967624 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:38.967628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:38.968033 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:38.968223 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:38.968236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.969617 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.969622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:38.970856 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:38.970874 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.971606 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.971611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:38.971772 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:38.972421 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:38.972439 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:38.975483 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":247,"threads_started":1}
W20260926 02:14:38.976029 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:38.966861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:49467 (local address 127.28.152.190:36985)
0926 02:14:38.967090 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:38.967097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:38.967236 (+ 139us) server_negotiation.cc:407] Connection header received
0926 02:14:38.967418 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:38.967424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:38.967497 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:38.967580 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:38.968387 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.969434 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.970980 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:38.971428 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:38.971532 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:38.975753 (+ 4221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:49467: BlockingRecv error: recv got EOF from 127.28.152.130:49467 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 02:14:39 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:39 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.131@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:39.484046 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:39.463388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.131:46481)
0926 02:14:39.464061 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:39.464077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:39.464189 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:39.464496 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:39.464500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:39.464899 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:39.465077 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:39.465087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.466087 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.466091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:39.467059 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:39.467072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.467764 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.467769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:39.467938 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:39.480397 (+ 12459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:39.480423 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:39.483824 (+ 3401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":462,"threads_started":1}
W20260926 02:14:39.484342 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:39.463566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:46481 (local address 127.28.152.190:36985)
0926 02:14:39.463732 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:39.463739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:39.464151 (+ 412us) server_negotiation.cc:407] Connection header received
0926 02:14:39.464270 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:39.464274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:39.464335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:39.464415 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:39.465231 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.465946 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.467196 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.467626 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.467719 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:39.484115 (+ 16396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.131:46481: BlockingRecv error: recv got EOF from 127.28.152.131:46481 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 02:14:39 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:39 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.129@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:39.530772 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:39.522146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.129:42305)
0926 02:14:39.522682 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:39.522699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:39.522844 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:39.523166 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:39.523168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:39.523593 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:39.523769 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:39.523834 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.524920 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.524925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:39.526099 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:39.526114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.526825 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.526830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:39.526946 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:39.527511 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:39.527528 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:39.530493 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":275,"threads_started":1}
W20260926 02:14:39.530965 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:39.522366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:42305 (local address 127.28.152.190:36985)
0926 02:14:39.522550 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:39.522555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:39.522837 (+ 282us) server_negotiation.cc:407] Connection header received
0926 02:14:39.522956 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:39.522960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:39.523029 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:39.523134 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:39.524004 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.524776 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.526248 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.526648 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.526777 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:39.530734 (+ 3957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.129:42305: BlockingRecv error: recv got EOF from 127.28.152.129:42305 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:14:39 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:14:39 dist-test-slave-hllj krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.152.130@KRBTEST.COM for oryx/127.28.152.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:14:39.985272 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:39.977032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36985 (local address 127.28.152.130:47193)
0926 02:14:39.977570 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:39.977588 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:39.977708 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:39.977994 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:39.977998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:39.978387 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:39.978552 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:39.978562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.979580 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.979584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:39.980510 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:39.980523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.981218 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.981222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:39.981351 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:39.981922 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:39.981942 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:39.984977 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.152.190:36985: Server oryx/127.28.152.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":305,"threads_started":1}
W20260926 02:14:39.985404 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:39.977213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:47193 (local address 127.28.152.190:36985)
0926 02:14:39.977422 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:39.977428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:39.977662 (+ 234us) server_negotiation.cc:407] Connection header received
0926 02:14:39.977795 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:39.977798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:39.977860 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:39.977939 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:39.978674 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.979451 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.980657 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:39.981082 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:39.981196 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:39.985201 (+ 4005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.152.130:47193: BlockingRecv error: recv got EOF from 127.28.152.130:47193 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:14:40.292052 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 5643
I20260926 02:14:40.304348 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:40.293204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.28.152.129:37793)
0926 02:14:40.293794 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:40.293801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:40.293829 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:14:40.293890 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:40.293897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:40.294117 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:40.294316 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:40.295748 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:40.296771 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:40.298122 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:40.298694 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:40.298804 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:40.301276 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:40.301286 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:40.301303 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:40.301338 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:40.303677 (+ 2339us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.152.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:14:40.304026 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56560: Request ticket server kudu/127.28.152.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
W20260926 02:14:40.304576 6106 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56560: Request ticket server kudu/127.28.152.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 02:14:40.304538 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 5643: Not authorized: Client connection negotiation failed: client connection to 127.28.152.129:37793: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.152.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:14:40.304718 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 5643
I20260926 02:14:40.318198 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 5794
I20260926 02:14:40.331146 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:40.319189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.28.152.130:44119)
0926 02:14:40.319776 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:40.319784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:40.319814 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:14:40.319873 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:40.319882 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:40.320112 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:40.320284 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:40.321637 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:40.322691 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:40.324004 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:40.324581 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:40.324700 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:40.327992 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:40.328003 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:40.328020 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:40.328057 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:40.330495 (+ 2438us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.152.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:14:40.330828 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40274: Request ticket server kudu/127.28.152.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":290,"threads_started":1}
W20260926 02:14:40.331260 6107 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40274: Request ticket server kudu/127.28.152.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 02:14:40.331283 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 5794: Not authorized: Client connection negotiation failed: client connection to 127.28.152.130:44119: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.152.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:14:40.331459 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 5794
I20260926 02:14:40.344669 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 5925
I20260926 02:14:40.356590 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:40.345661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.28.152.131:38995)
0926 02:14:40.346305 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:40.346311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:40.346335 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:14:40.346396 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:40.346403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:40.346615 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:40.346777 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:40.348069 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:40.349071 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:40.350437 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:40.350959 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:40.351051 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:40.353602 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:40.353611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:40.353626 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:40.353658 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:40.356091 (+ 2433us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.152.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:14:40.356363 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47686: Request ticket server kudu/127.28.152.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":386,"threads_started":1}
W20260926 02:14:40.356698 6108 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47686: Request ticket server kudu/127.28.152.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 02:14:40.356866 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 5925: Not authorized: Client connection negotiation failed: client connection to 127.28.152.131:38995: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.152.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:14:40.356999 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 5925
I20260926 02:14:40.370973 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 5574
I20260926 02:14:40.372280 5636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:40.484248 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 5574
2026-09-26T02:14:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20964 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T02:14:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:14:40Z Disabled control of system clock
I20260926 02:14:40.555490 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:36073
--webserver_interface=127.28.152.190
--webserver_port=0
--builtin_ntp_servers=127.28.152.148:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:36073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 02:14:40.884021 6113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:40.884330 6113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:40.884406 6113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:40.891764 6113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:14:40.891896 6113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:40.891935 6113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:40.891984 6113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:14:40.892027 6113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:14:40.902573 6113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:36073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:36073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:40.903905 6113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:40.905794 6113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:40.918483 6119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:40.918560 6121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:40.918530 6118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:40.918958 6113 server_base.cc:1034] running on GCE node
I20260926 02:14:40.920017 6113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:40.921686 6113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:40.922945 6113 hybrid_clock.cc:648] HybridClock initialized: now 1790388880922892 us; error 50 us; skew 500 ppm
I20260926 02:14:40.923347 6113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:14:40.925951 6113 webserver.cc:460] Webserver started at http://127.28.152.190:38677/ using document root <none> and password file <none>
I20260926 02:14:40.926734 6113 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:40.926839 6113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:40.927134 6113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:40.930266 6113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "b75b711fcf5e4e27863a749c21b68e69"
format_stamp: "Formatted at 2026-09-26 02:14:40 on dist-test-slave-hllj"
server_key: "b52eacfeedfbc12bec311d01dc5ae0c8"
server_key_iv: "6469b5d75060b96c58395df08c448882"
server_key_version: "encryptionkey@0"
I20260926 02:14:40.931105 6113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "b75b711fcf5e4e27863a749c21b68e69"
format_stamp: "Formatted at 2026-09-26 02:14:40 on dist-test-slave-hllj"
server_key: "b52eacfeedfbc12bec311d01dc5ae0c8"
server_key_iv: "6469b5d75060b96c58395df08c448882"
server_key_version: "encryptionkey@0"
I20260926 02:14:40.936884 6113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:14:40.940769 6127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:40.942384 6113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:14:40.942572 6113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "b75b711fcf5e4e27863a749c21b68e69"
format_stamp: "Formatted at 2026-09-26 02:14:40 on dist-test-slave-hllj"
server_key: "b52eacfeedfbc12bec311d01dc5ae0c8"
server_key_iv: "6469b5d75060b96c58395df08c448882"
server_key_version: "encryptionkey@0"
I20260926 02:14:40.942797 6113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:40.983143 6113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:40.984077 6113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:40.984366 6113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:41.001613 6113 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:36073
I20260926 02:14:41.001621 6178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:36073 every 8 connection(s)
I20260926 02:14:41.002969 6113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:14:41.006819 6179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:41.007813 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 6113
I20260926 02:14:41.007997 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:14:41.008575 29282 external_mini_cluster.cc:1428] Setting key 9f0486d4c7d1eb01c61b372bf670cae2
I20260926 02:14:41.018860 6179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69: Bootstrap starting.
I20260926 02:14:41.022493 6179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:41.023272 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:41.010750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.28.152.190:36073)
0926 02:14:41.011859 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:41.011874 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:41.011920 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:14:41.013672 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:41.013697 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:41.014234 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:41.014589 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:41.015361 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:41.016756 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:41.018712 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:41.019460 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:41.019592 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:41.020841 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:41.020850 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:41.020866 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:41.021458 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:41.021537 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:41.021559 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:41.021645 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:41.021905 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 02:14:41.022241 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":378,"threads_started":1}
I20260926 02:14:41.023643 6179 log.cc:824] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:41.027052 6179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69: No bootstrap required, opened a new log
I20260926 02:14:41.034559 6179 raft_consensus.cc:348] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b711fcf5e4e27863a749c21b68e69" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36073 } }
I20260926 02:14:41.034957 6179 raft_consensus.cc:374] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:41.035048 6179 raft_consensus.cc:729] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75b711fcf5e4e27863a749c21b68e69, State: Initialized, Role: FOLLOWER
I20260926 02:14:41.035621 6179 consensus_queue.cc:260] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b711fcf5e4e27863a749c21b68e69" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36073 } }
I20260926 02:14:41.035830 6179 raft_consensus.cc:388] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:14:41.035951 6179 raft_consensus.cc:482] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:14:41.036101 6179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:41.038786 6179 raft_consensus.cc:504] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b711fcf5e4e27863a749c21b68e69" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36073 } }
I20260926 02:14:41.039317 6179 leader_election.cc:302] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75b711fcf5e4e27863a749c21b68e69; no voters:
I20260926 02:14:41.039897 6179 leader_election.cc:288] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:14:41.040027 6184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:41.040674 6184 raft_consensus.cc:686] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [term 1 LEADER]: Becoming Leader. State: Replica: b75b711fcf5e4e27863a749c21b68e69, State: Running, Role: LEADER
I20260926 02:14:41.041277 6184 consensus_queue.cc:237] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b711fcf5e4e27863a749c21b68e69" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36073 } }
I20260926 02:14:41.041956 6179 sys_catalog.cc:564] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:14:41.043982 6186 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b75b711fcf5e4e27863a749c21b68e69. Latest consensus state: current_term: 1 leader_uuid: "b75b711fcf5e4e27863a749c21b68e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b711fcf5e4e27863a749c21b68e69" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36073 } } }
I20260926 02:14:41.044416 6186 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:41.044338 6185 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b75b711fcf5e4e27863a749c21b68e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75b711fcf5e4e27863a749c21b68e69" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36073 } } }
I20260926 02:14:41.044634 6185 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:41.045262 6191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:14:41.050665 6191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:14:41.059314 6191 catalog_manager.cc:1347] Generated new cluster ID: e32b4deb8f8d4f69bc5c1f7213f82ce1
I20260926 02:14:41.059444 6191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:14:41.074626 6191 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:14:41.075891 6191 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:14:41.084530 6191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b75b711fcf5e4e27863a749c21b68e69: Generated new TSK 0
I20260926 02:14:41.085445 6191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:14:41.098582 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36073
--builtin_ntp_servers=127.28.152.148:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 02:14:41.432381 6203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:41.432685 6203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:41.432796 6203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:41.440169 6203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:41.440303 6203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:41.440448 6203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:14:41.450476 6203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:41.451784 6203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:41.453814 6203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:41.466885 6208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:41.466922 6211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:41.466954 6209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:41.467779 6203 server_base.cc:1034] running on GCE node
I20260926 02:14:41.468398 6203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:41.469431 6203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:41.470664 6203 hybrid_clock.cc:648] HybridClock initialized: now 1790388881470596 us; error 51 us; skew 500 ppm
I20260926 02:14:41.471027 6203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:14:41.473603 6203 webserver.cc:460] Webserver started at http://127.28.152.129:40339/ using document root <none> and password file <none>
I20260926 02:14:41.474366 6203 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:41.474464 6203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:41.474777 6203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:41.477919 6203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "3886324d68074c138e701e6d4a9f2420"
format_stamp: "Formatted at 2026-09-26 02:14:41 on dist-test-slave-hllj"
server_key: "4b61fe5d7bc2e2cf33922707b1d720f6"
server_key_iv: "7f24ea44f6ee8bcaf7677b4703a9b74b"
server_key_version: "encryptionkey@0"
I20260926 02:14:41.478834 6203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "3886324d68074c138e701e6d4a9f2420"
format_stamp: "Formatted at 2026-09-26 02:14:41 on dist-test-slave-hllj"
server_key: "4b61fe5d7bc2e2cf33922707b1d720f6"
server_key_iv: "7f24ea44f6ee8bcaf7677b4703a9b74b"
server_key_version: "encryptionkey@0"
I20260926 02:14:41.484467 6203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:14:41.488627 6217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:41.490212 6203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:14:41.490427 6203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "3886324d68074c138e701e6d4a9f2420"
format_stamp: "Formatted at 2026-09-26 02:14:41 on dist-test-slave-hllj"
server_key: "4b61fe5d7bc2e2cf33922707b1d720f6"
server_key_iv: "7f24ea44f6ee8bcaf7677b4703a9b74b"
server_key_version: "encryptionkey@0"
I20260926 02:14:41.490653 6203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:41.507002 6203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:41.507926 6203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:41.508204 6203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:41.509189 6203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:41.510653 6203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:41.510739 6203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:41.510850 6203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:41.510921 6203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:41.541859 6203 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:33657
I20260926 02:14:41.541949 6330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:33657 every 8 connection(s)
I20260926 02:14:41.543505 6203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:14:41.552515 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 6203
I20260926 02:14:41.552727 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:14:41.553624 29282 external_mini_cluster.cc:1428] Setting key 614bd47751e8c8e519b80d2d9bfd0adc
I20260926 02:14:41.556661 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36073
--builtin_ntp_servers=127.28.152.148:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 02:14:41.557981 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:41.546382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:58069 (local address 127.28.152.190:36073)
0926 02:14:41.546890 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:41.546894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:41.548282 (+ 1388us) server_negotiation.cc:407] Connection header received
0926 02:14:41.550308 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:41.550313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:41.550404 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:41.550576 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:41.551830 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:41.552562 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:41.555005 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:41.555434 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:41.555563 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:41.557094 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:41.557102 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:41.557107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:41.557152 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:41.557208 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:41.557226 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:41.557313 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:41.557526 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 02:14:41.557636 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":293,"threads_started":1}
I20260926 02:14:41.559034 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:41.546998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36073 (local address 127.28.152.129:58069)
0926 02:14:41.548111 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:41.548164 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:41.549970 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:41.550867 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:41.550877 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:41.550991 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:41.551644 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:41.551663 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:41.552777 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:41.552783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:41.554839 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:41.554857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:41.555679 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:41.555687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:41.555942 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:41.556690 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:14:41.556693 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:41.556740 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:14:41.556844 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:41.557312 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:41.557320 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:41.557326 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:14:41.557500 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 02:14:41.557848 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":305,"threads_started":1}
I20260926 02:14:41.560803 6331 heartbeater.cc:347] Connected to a master server at 127.28.152.190:36073
I20260926 02:14:41.561161 6331 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:41.561987 6331 heartbeater.cc:511] Master 127.28.152.190:36073 requested a full tablet report, sending...
I20260926 02:14:41.564184 6144 ts_manager.cc:194] Registered new tserver with Master: 3886324d68074c138e701e6d4a9f2420 (127.28.152.129:33657)
I20260926 02:14:41.565824 6144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.129:58069
W20260926 02:14:41.888386 6335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:41.888693 6335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:41.888830 6335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:41.896987 6335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:41.897145 6335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:41.897305 6335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:14:41.908296 6335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:41.909744 6335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:41.911550 6335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:41.925279 6340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:41.925221 6341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:41.925252 6343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:41.926607 6335 server_base.cc:1034] running on GCE node
I20260926 02:14:41.927356 6335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:41.928478 6335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:41.929702 6335 hybrid_clock.cc:648] HybridClock initialized: now 1790388881929642 us; error 47 us; skew 500 ppm
I20260926 02:14:41.930056 6335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:14:41.932664 6335 webserver.cc:460] Webserver started at http://127.28.152.130:38669/ using document root <none> and password file <none>
I20260926 02:14:41.933481 6335 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:41.933593 6335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:41.933920 6335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:41.936874 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "0bd7ed5787b343fb93b01369865139d9"
format_stamp: "Formatted at 2026-09-26 02:14:41 on dist-test-slave-hllj"
server_key: "f0bca0ae942831c6bb906677710b88d7"
server_key_iv: "82cecc40fdbc0c9ca473016a0e720553"
server_key_version: "encryptionkey@0"
I20260926 02:14:41.937824 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "0bd7ed5787b343fb93b01369865139d9"
format_stamp: "Formatted at 2026-09-26 02:14:41 on dist-test-slave-hllj"
server_key: "f0bca0ae942831c6bb906677710b88d7"
server_key_iv: "82cecc40fdbc0c9ca473016a0e720553"
server_key_version: "encryptionkey@0"
I20260926 02:14:41.943760 6335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 02:14:41.947896 6349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:41.949699 6335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:14:41.949900 6335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "0bd7ed5787b343fb93b01369865139d9"
format_stamp: "Formatted at 2026-09-26 02:14:41 on dist-test-slave-hllj"
server_key: "f0bca0ae942831c6bb906677710b88d7"
server_key_iv: "82cecc40fdbc0c9ca473016a0e720553"
server_key_version: "encryptionkey@0"
I20260926 02:14:41.950124 6335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:41.968545 6335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:41.969551 6335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:41.969857 6335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:41.970875 6335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:41.972524 6335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:41.972616 6335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:41.972738 6335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:41.972807 6335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:42.003829 6335 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:38593
I20260926 02:14:42.003849 6462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:38593 every 8 connection(s)
I20260926 02:14:42.005200 6335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:14:42.009972 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 6335
I20260926 02:14:42.010166 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:14:42.010576 29282 external_mini_cluster.cc:1428] Setting key da968a84be021bec91ba4c5d5b21a2fd
I20260926 02:14:42.013304 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36073
--builtin_ntp_servers=127.28.152.148:35205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 02:14:42.020460 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:42.007967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:39597 (local address 127.28.152.190:36073)
0926 02:14:42.008167 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:42.008173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:42.010056 (+ 1883us) server_negotiation.cc:407] Connection header received
0926 02:14:42.012530 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:42.012535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:42.012613 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:42.012721 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:42.014186 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.015211 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.017597 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.018157 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.018291 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:42.019759 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:42.019765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:42.019768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:42.019789 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:42.019828 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:42.019840 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:42.019924 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:42.020140 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 02:14:42.020225 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:14:42.022125 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:42.008709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36073 (local address 127.28.152.130:39597)
0926 02:14:42.009848 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:42.009898 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:42.012212 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:42.013001 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:42.013013 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:42.013183 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:42.013994 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:42.014019 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.015371 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.015377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:42.017430 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:42.017446 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.018406 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.018414 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:42.018654 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:42.019350 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:14:42.019354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:42.019418 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:14:42.019531 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:42.019927 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:42.019937 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:42.019944 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:14:42.020126 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 02:14:42.020715 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":328,"threads_started":1}
I20260926 02:14:42.023638 6463 heartbeater.cc:347] Connected to a master server at 127.28.152.190:36073
I20260926 02:14:42.023938 6463 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:42.024746 6463 heartbeater.cc:511] Master 127.28.152.190:36073 requested a full tablet report, sending...
I20260926 02:14:42.026266 6144 ts_manager.cc:194] Registered new tserver with Master: 0bd7ed5787b343fb93b01369865139d9 (127.28.152.130:38593)
I20260926 02:14:42.027093 6144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.130:39597
W20260926 02:14:42.347306 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:42.347641 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:42.347770 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:42.355324 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:42.355449 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:42.355592 6466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:14:42.364894 6466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:36073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:42.366245 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:42.368129 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:42.381549 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:42.381595 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:42.381781 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:42.382673 6466 server_base.cc:1034] running on GCE node
I20260926 02:14:42.383288 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:42.384316 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:42.385566 6466 hybrid_clock.cc:648] HybridClock initialized: now 1790388882385495 us; error 59 us; skew 500 ppm
I20260926 02:14:42.385983 6466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:14:42.388590 6466 webserver.cc:460] Webserver started at http://127.28.152.131:43789/ using document root <none> and password file <none>
I20260926 02:14:42.389496 6466 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:42.389606 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:42.389925 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:42.392985 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "d9dc78bb9d4349d48143ef8f04613d3c"
format_stamp: "Formatted at 2026-09-26 02:14:42 on dist-test-slave-hllj"
server_key: "0ead45327ded9b1625fdffb5cdcd57e8"
server_key_iv: "e83fdfb5ff2a2239ced6c3cfcdcc3790"
server_key_version: "encryptionkey@0"
I20260926 02:14:42.393929 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "d9dc78bb9d4349d48143ef8f04613d3c"
format_stamp: "Formatted at 2026-09-26 02:14:42 on dist-test-slave-hllj"
server_key: "0ead45327ded9b1625fdffb5cdcd57e8"
server_key_iv: "e83fdfb5ff2a2239ced6c3cfcdcc3790"
server_key_version: "encryptionkey@0"
I20260926 02:14:42.399585 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 02:14:42.403491 6480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:42.404987 6466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:14:42.405210 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "d9dc78bb9d4349d48143ef8f04613d3c"
format_stamp: "Formatted at 2026-09-26 02:14:42 on dist-test-slave-hllj"
server_key: "0ead45327ded9b1625fdffb5cdcd57e8"
server_key_iv: "e83fdfb5ff2a2239ced6c3cfcdcc3790"
server_key_version: "encryptionkey@0"
I20260926 02:14:42.405441 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:42.438971 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:42.439901 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:42.440186 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:42.441231 6466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:42.442764 6466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:42.442873 6466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:42.442998 6466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:42.443080 6466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:42.473425 6466 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:39593
I20260926 02:14:42.473498 6594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:39593 every 8 connection(s)
I20260926 02:14:42.474828 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:14:42.477468 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 6466
I20260926 02:14:42.477638 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:14:42.478075 29282 external_mini_cluster.cc:1428] Setting key 24876f1857c7b13c0fd7d59fe7e77dc2
I20260926 02:14:42.487527 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:42.477463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:41495 (local address 127.28.152.190:36073)
0926 02:14:42.477669 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:42.477675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:42.479291 (+ 1616us) server_negotiation.cc:407] Connection header received
0926 02:14:42.481189 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:42.481193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:42.481260 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:42.481338 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:42.482461 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.483183 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.484766 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.485291 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.485397 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:42.486779 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:42.486784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:42.486788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:42.486810 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:42.486873 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:42.486886 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:42.486965 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:42.487186 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 02:14:42.487271 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:14:42.488653 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:42.478035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:36073 (local address 127.28.152.131:41495)
0926 02:14:42.479127 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:42.479183 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:42.480896 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:42.481570 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:42.481579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:42.481678 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:42.482300 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:42.482323 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.483333 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.483337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:42.484623 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:42.484635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.485536 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.485541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:42.485764 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:42.486444 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:14:42.486447 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:42.486490 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:14:42.486575 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:42.486987 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:42.486996 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:42.487002 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:14:42.487158 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 02:14:42.487516 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":324,"threads_started":1}
I20260926 02:14:42.490231 6595 heartbeater.cc:347] Connected to a master server at 127.28.152.190:36073
I20260926 02:14:42.490594 6595 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:42.491488 6595 heartbeater.cc:511] Master 127.28.152.190:36073 requested a full tablet report, sending...
I20260926 02:14:42.493237 6144 ts_manager.cc:194] Registered new tserver with Master: d9dc78bb9d4349d48143ef8f04613d3c (127.28.152.131:39593)
I20260926 02:14:42.494159 6144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.131:41495
I20260926 02:14:42.494179 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 02:14:42.500597 6333 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:42.499105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.28.152.190:36073)
0926 02:14:42.499277 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:42.499281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:42.499717 (+ 436us) server_negotiation.cc:407] Connection header received
0926 02:14:42.499884 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:42.499887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:42.499990 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:42.500073 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:42.500394 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38354: BlockingRecv error: recv got EOF from 127.0.0.1:38354 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:14:42.501379 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 6203
I20260926 02:14:42.509198 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:42.502270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.28.152.129:33657)
0926 02:14:42.502754 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:42.502762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:42.502789 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:14:42.502853 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:42.502860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:42.503091 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:42.503215 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:42.503559 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.504566 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.505904 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.506485 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.506583 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:42.507613 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:42.507622 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:42.507637 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:42.508260 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:42.508309 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:42.508329 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:42.508433 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:42.508622 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 02:14:42.508743 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":212,"threads_started":1}
I20260926 02:14:42.510571 6326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:42.590806 6331 heartbeater.cc:503] Master 127.28.152.190:36073 was elected leader, sending a full tablet report...
I20260926 02:14:42.591995 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 6203
W20260926 02:14:42.604854 6129 connection.cc:535] server connection from 127.28.152.129:58069 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 02:14:42.605198 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 6335
I20260926 02:14:42.613282 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:42.606121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.28.152.130:38593)
0926 02:14:42.606625 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:42.606632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:42.606657 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:14:42.606719 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:42.606737 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:42.606953 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:42.607093 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:42.607456 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.608451 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.609957 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.610637 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.610748 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:42.611815 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:42.611824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:42.611839 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:42.612471 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:42.612523 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:42.612542 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:42.612627 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:42.612772 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 02:14:42.612902 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":251,"threads_started":1}
I20260926 02:14:42.614571 6458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:42.695770 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 6335
I20260926 02:14:42.708554 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 6466
I20260926 02:14:42.716724 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:42.709565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.28.152.131:39593)
0926 02:14:42.710238 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:42.710245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:42.710275 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:14:42.710334 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:42.710343 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:42.710554 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:42.710711 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:42.711144 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.712182 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.713579 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:42.714142 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:42.714252 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:42.715320 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:42.715327 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:42.715343 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:42.715929 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:42.715986 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:42.716006 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:42.716105 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:42.716253 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 02:14:42.716385 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":378,"threads_started":1}
I20260926 02:14:42.717937 6590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:42.799064 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 6466
I20260926 02:14:42.812003 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 6113
I20260926 02:14:42.813122 6174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:42.913355 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 6113
2026-09-26T02:14:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2424 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T02:14:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:14:42Z Disabled control of system clock
I20260926 02:14:42.979630 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:41439
--webserver_interface=127.28.152.190
--webserver_port=0
--builtin_ntp_servers=127.28.152.148:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 02:14:43.304217 6614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:43.304543 6614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:43.304654 6614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:43.312224 6614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:14:43.312340 6614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:43.312382 6614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:43.312459 6614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:14:43.312511 6614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:14:43.322054 6614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:41439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:43.323385 6614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:43.325268 6614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:43.337378 6619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:43.337456 6620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:43.337407 6622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:43.338229 6614 server_base.cc:1034] running on GCE node
I20260926 02:14:43.339056 6614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:43.340619 6614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:43.341825 6614 hybrid_clock.cc:648] HybridClock initialized: now 1790388883341772 us; error 49 us; skew 500 ppm
I20260926 02:14:43.342202 6614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:14:43.344518 6614 webserver.cc:460] Webserver started at http://127.28.152.190:40923/ using document root <none> and password file <none>
I20260926 02:14:43.345280 6614 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:43.345388 6614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:43.345705 6614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:43.348642 6614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "ae20b51e68604f23b85812c00628c918"
format_stamp: "Formatted at 2026-09-26 02:14:43 on dist-test-slave-hllj"
server_key: "f09b2541e14fd23a9c59ec2081bec8d4"
server_key_iv: "76f6707a96d26471e18d9a26439cbdbf"
server_key_version: "encryptionkey@0"
I20260926 02:14:43.349509 6614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "ae20b51e68604f23b85812c00628c918"
format_stamp: "Formatted at 2026-09-26 02:14:43 on dist-test-slave-hllj"
server_key: "f09b2541e14fd23a9c59ec2081bec8d4"
server_key_iv: "76f6707a96d26471e18d9a26439cbdbf"
server_key_version: "encryptionkey@0"
I20260926 02:14:43.354818 6614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:14:43.358407 6628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:43.359792 6614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:14:43.359964 6614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "ae20b51e68604f23b85812c00628c918"
format_stamp: "Formatted at 2026-09-26 02:14:43 on dist-test-slave-hllj"
server_key: "f09b2541e14fd23a9c59ec2081bec8d4"
server_key_iv: "76f6707a96d26471e18d9a26439cbdbf"
server_key_version: "encryptionkey@0"
I20260926 02:14:43.360165 6614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:43.366143 6614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:43.367022 6614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:43.367311 6614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:43.385468 6614 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:41439
I20260926 02:14:43.385514 6679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:41439 every 8 connection(s)
I20260926 02:14:43.386796 6614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:14:43.390646 6680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:43.391178 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 6614
I20260926 02:14:43.391346 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:14:43.391970 29282 external_mini_cluster.cc:1428] Setting key dab10f6bcb65f810b673c60aab94e2fe
I20260926 02:14:43.401497 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:43.394103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.28.152.190:41439)
0926 02:14:43.395283 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:43.395300 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:43.395357 (+ 57us) server_negotiation.cc:407] Connection header received
0926 02:14:43.397170 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:43.397203 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:43.397861 (+ 658us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:43.398350 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:43.399356 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:43.399365 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:43.399383 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:43.399402 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:43.399485 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:43.399508 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:43.399719 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:43.399878 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 02:14:43.400182 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":341,"threads_started":1}
I20260926 02:14:43.405002 6680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918: Bootstrap starting.
I20260926 02:14:43.408066 6680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:43.409108 6680 log.cc:824] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:43.412297 6680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918: No bootstrap required, opened a new log
I20260926 02:14:43.420264 6680 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae20b51e68604f23b85812c00628c918" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41439 } }
I20260926 02:14:43.420680 6680 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:43.420758 6680 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae20b51e68604f23b85812c00628c918, State: Initialized, Role: FOLLOWER
I20260926 02:14:43.421443 6680 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae20b51e68604f23b85812c00628c918" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41439 } }
I20260926 02:14:43.421674 6680 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:14:43.421777 6680 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:14:43.421906 6680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:43.424366 6680 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae20b51e68604f23b85812c00628c918" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41439 } }
I20260926 02:14:43.424868 6680 leader_election.cc:302] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae20b51e68604f23b85812c00628c918; no voters:
I20260926 02:14:43.425513 6680 leader_election.cc:288] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:14:43.425629 6685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:43.426265 6685 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [term 1 LEADER]: Becoming Leader. State: Replica: ae20b51e68604f23b85812c00628c918, State: Running, Role: LEADER
I20260926 02:14:43.426813 6685 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae20b51e68604f23b85812c00628c918" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41439 } }
I20260926 02:14:43.427591 6680 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:14:43.429574 6687 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae20b51e68604f23b85812c00628c918. Latest consensus state: current_term: 1 leader_uuid: "ae20b51e68604f23b85812c00628c918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae20b51e68604f23b85812c00628c918" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41439 } } }
I20260926 02:14:43.430002 6687 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:43.430246 6686 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae20b51e68604f23b85812c00628c918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae20b51e68604f23b85812c00628c918" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41439 } } }
I20260926 02:14:43.430687 6686 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:43.432701 6691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:14:43.438110 6691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:14:43.448545 6691 catalog_manager.cc:1347] Generated new cluster ID: c05ca4c4f2c54ba291b0d19ca58d9b3d
I20260926 02:14:43.448688 6691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:14:43.462499 6691 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:14:43.462904 6691 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:14:43.473978 6691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae20b51e68604f23b85812c00628c918: Generated new TSK 0
I20260926 02:14:43.474831 6691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:14:43.486301 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41439
--builtin_ntp_servers=127.28.152.148:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 02:14:43.836287 6704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:43.836632 6704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:43.836760 6704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:43.845237 6704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:43.845377 6704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:43.845559 6704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:14:43.855880 6704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:43.857259 6704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:43.859099 6704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:43.872023 6709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:43.872105 6710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:43.872143 6712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:43.873013 6704 server_base.cc:1034] running on GCE node
I20260926 02:14:43.873689 6704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:43.874679 6704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:43.875867 6704 hybrid_clock.cc:648] HybridClock initialized: now 1790388883875803 us; error 58 us; skew 500 ppm
I20260926 02:14:43.876174 6704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:14:43.878571 6704 webserver.cc:460] Webserver started at http://127.28.152.129:44457/ using document root <none> and password file <none>
I20260926 02:14:43.879315 6704 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:43.879406 6704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:43.879727 6704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:43.884408 6704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "037d3f984911407d8e7236b81cbd484b"
format_stamp: "Formatted at 2026-09-26 02:14:43 on dist-test-slave-hllj"
server_key: "bf79d6395d193d85ca544b9081d552ee"
server_key_iv: "434d39ca6e9ecafa098b5443124c1a19"
server_key_version: "encryptionkey@0"
I20260926 02:14:43.885440 6704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "037d3f984911407d8e7236b81cbd484b"
format_stamp: "Formatted at 2026-09-26 02:14:43 on dist-test-slave-hllj"
server_key: "bf79d6395d193d85ca544b9081d552ee"
server_key_iv: "434d39ca6e9ecafa098b5443124c1a19"
server_key_version: "encryptionkey@0"
I20260926 02:14:43.891294 6704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260926 02:14:43.895082 6719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:43.896531 6704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:14:43.896696 6704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "037d3f984911407d8e7236b81cbd484b"
format_stamp: "Formatted at 2026-09-26 02:14:43 on dist-test-slave-hllj"
server_key: "bf79d6395d193d85ca544b9081d552ee"
server_key_iv: "434d39ca6e9ecafa098b5443124c1a19"
server_key_version: "encryptionkey@0"
I20260926 02:14:43.896958 6704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:43.902941 6704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:43.903775 6704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:43.904036 6704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:43.904954 6704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:43.906500 6704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:43.906607 6704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:43.906715 6704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:43.906806 6704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:43.938760 6704 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:38375
I20260926 02:14:43.938864 6831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:38375 every 8 connection(s)
I20260926 02:14:43.940070 6704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:14:43.945737 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 6704
I20260926 02:14:43.945935 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:14:43.946714 29282 external_mini_cluster.cc:1428] Setting key 9553fc13773317afe07e61baabff78c4
I20260926 02:14:43.949203 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:43.942666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:49285 (local address 127.28.152.190:41439)
0926 02:14:43.943131 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:43.943135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:43.944458 (+ 1323us) server_negotiation.cc:407] Connection header received
0926 02:14:43.946730 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:43.946735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:43.946807 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:43.946928 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:43.948407 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:43.948413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:43.948417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:43.948437 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:43.948493 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:43.948510 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:43.948642 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:43.948878 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 02:14:43.948935 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":280,"threads_started":1}
I20260926 02:14:43.949803 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41439
--builtin_ntp_servers=127.28.152.148:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 02:14:43.950589 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:43.943306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:41439 (local address 127.28.152.129:49285)
0926 02:14:43.944270 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:43.944317 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:43.946392 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:43.947223 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:43.947233 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:43.947341 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:43.948096 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:14:43.948099 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:43.948141 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:14:43.948236 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:43.948617 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:43.948625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:43.948631 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:14:43.948926 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 02:14:43.949278 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":279,"threads_started":1}
I20260926 02:14:43.952346 6832 heartbeater.cc:347] Connected to a master server at 127.28.152.190:41439
I20260926 02:14:43.952667 6832 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:43.953378 6832 heartbeater.cc:511] Master 127.28.152.190:41439 requested a full tablet report, sending...
I20260926 02:14:43.955554 6645 ts_manager.cc:194] Registered new tserver with Master: 037d3f984911407d8e7236b81cbd484b (127.28.152.129:38375)
W20260926 02:14:44.279630 6836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:44.279955 6836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:44.280092 6836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:44.287489 6836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:44.287616 6836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:44.287760 6836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:14:44.297081 6836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:44.298475 6836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:44.300340 6836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:44.314388 6842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:44.314355 6841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:44.314545 6844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:44.315165 6836 server_base.cc:1034] running on GCE node
I20260926 02:14:44.315790 6836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:44.316798 6836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:44.317986 6836 hybrid_clock.cc:648] HybridClock initialized: now 1790388884317905 us; error 72 us; skew 500 ppm
I20260926 02:14:44.318282 6836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:14:44.320622 6836 webserver.cc:460] Webserver started at http://127.28.152.130:44255/ using document root <none> and password file <none>
I20260926 02:14:44.321625 6836 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:44.321753 6836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:44.322130 6836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:44.326270 6836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "d7382a818c4f469aa306159773f34cf0"
format_stamp: "Formatted at 2026-09-26 02:14:44 on dist-test-slave-hllj"
server_key: "12dd89835a61badab120c402953796c8"
server_key_iv: "96f57336a2fb29f003c3ac07e64b1855"
server_key_version: "encryptionkey@0"
I20260926 02:14:44.327368 6836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "d7382a818c4f469aa306159773f34cf0"
format_stamp: "Formatted at 2026-09-26 02:14:44 on dist-test-slave-hllj"
server_key: "12dd89835a61badab120c402953796c8"
server_key_iv: "96f57336a2fb29f003c3ac07e64b1855"
server_key_version: "encryptionkey@0"
I20260926 02:14:44.333292 6836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:14:44.337179 6850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:44.338738 6836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:14:44.338914 6836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "d7382a818c4f469aa306159773f34cf0"
format_stamp: "Formatted at 2026-09-26 02:14:44 on dist-test-slave-hllj"
server_key: "12dd89835a61badab120c402953796c8"
server_key_iv: "96f57336a2fb29f003c3ac07e64b1855"
server_key_version: "encryptionkey@0"
I20260926 02:14:44.339149 6836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:44.345376 6836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:44.346282 6836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:44.346551 6836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:44.347577 6836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:44.349051 6836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:44.349200 6836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:44.349324 6836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:44.349391 6836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:44.379354 6836 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:46401
I20260926 02:14:44.379424 6964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:46401 every 8 connection(s)
I20260926 02:14:44.380661 6836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:14:44.382280 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 6836
I20260926 02:14:44.382462 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:14:44.383080 29282 external_mini_cluster.cc:1428] Setting key 38f7a3a9704b90f09b0aee28bf1dbce2
I20260926 02:14:44.386194 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41439
--builtin_ntp_servers=127.28.152.148:44451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 02:14:44.390463 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:44.383425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:47645 (local address 127.28.152.190:41439)
0926 02:14:44.383625 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:44.383631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:44.385547 (+ 1916us) server_negotiation.cc:407] Connection header received
0926 02:14:44.387857 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:44.387863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:44.387945 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:44.388042 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:44.389738 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:44.389745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:44.389750 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:44.389769 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:44.389811 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:44.389828 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:44.389943 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:44.390132 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 02:14:44.390190 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:14:44.391842 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:44.384185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:41439 (local address 127.28.152.130:47645)
0926 02:14:44.385387 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:44.385447 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:44.387533 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:44.388444 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:44.388468 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:44.388600 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:44.389361 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:14:44.389365 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:44.389420 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:14:44.389550 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:44.389932 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:44.389942 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:44.389950 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:14:44.390209 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 02:14:44.390486 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":297,"threads_started":1}
I20260926 02:14:44.393405 6965 heartbeater.cc:347] Connected to a master server at 127.28.152.190:41439
I20260926 02:14:44.393756 6965 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:44.394474 6965 heartbeater.cc:511] Master 127.28.152.190:41439 requested a full tablet report, sending...
I20260926 02:14:44.395907 6645 ts_manager.cc:194] Registered new tserver with Master: d7382a818c4f469aa306159773f34cf0 (127.28.152.130:46401)
W20260926 02:14:44.719973 6968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:44.720278 6968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:44.720395 6968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:44.727870 6968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:44.727990 6968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:44.728132 6968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:14:44.737841 6968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:44.739198 6968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:44.741010 6968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:44.754622 6974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:44.754694 6976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:44.754604 6973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:44.755771 6968 server_base.cc:1034] running on GCE node
I20260926 02:14:44.756407 6968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:44.757521 6968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:44.758750 6968 hybrid_clock.cc:648] HybridClock initialized: now 1790388884758677 us; error 69 us; skew 500 ppm
I20260926 02:14:44.759104 6968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:14:44.766932 6968 webserver.cc:460] Webserver started at http://127.28.152.131:39539/ using document root <none> and password file <none>
I20260926 02:14:44.767758 6968 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:44.767861 6968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:44.768188 6968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:44.771283 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "e946f757561c4d1c8938ae6c6ae7ed86"
format_stamp: "Formatted at 2026-09-26 02:14:44 on dist-test-slave-hllj"
server_key: "2d0907061d523582321d9c580cd5ee74"
server_key_iv: "cefb89a81a29caaa9ef389407223e7f0"
server_key_version: "encryptionkey@0"
I20260926 02:14:44.772104 6968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "e946f757561c4d1c8938ae6c6ae7ed86"
format_stamp: "Formatted at 2026-09-26 02:14:44 on dist-test-slave-hllj"
server_key: "2d0907061d523582321d9c580cd5ee74"
server_key_iv: "cefb89a81a29caaa9ef389407223e7f0"
server_key_version: "encryptionkey@0"
I20260926 02:14:44.777570 6968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 02:14:44.781339 6982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:44.782889 6968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:14:44.783074 6968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "e946f757561c4d1c8938ae6c6ae7ed86"
format_stamp: "Formatted at 2026-09-26 02:14:44 on dist-test-slave-hllj"
server_key: "2d0907061d523582321d9c580cd5ee74"
server_key_iv: "cefb89a81a29caaa9ef389407223e7f0"
server_key_version: "encryptionkey@0"
I20260926 02:14:44.783311 6968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:44.789412 6968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:44.790289 6968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:44.790552 6968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:44.791491 6968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:44.792941 6968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:44.793041 6968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:44.793176 6968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:44.793251 6968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 02:14:44.822003 6968 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:34221
I20260926 02:14:44.822053 7095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:34221 every 8 connection(s)
I20260926 02:14:44.823390 6968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:14:44.831514 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:44.826044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:41909 (local address 127.28.152.190:41439)
0926 02:14:44.826220 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:44.826224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:44.827735 (+ 1511us) server_negotiation.cc:407] Connection header received
0926 02:14:44.829542 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:44.829547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:44.829601 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:44.829700 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:44.830944 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:44.830948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:44.830951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:44.830968 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:44.830998 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:44.831009 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:44.831101 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:44.831232 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 02:14:44.831279 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:14:44.832445 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:44.826611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:41439 (local address 127.28.152.131:41909)
0926 02:14:44.827574 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:44.827620 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:44.829282 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:44.829906 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:44.829914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:44.830047 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:44.830663 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:14:44.830665 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:44.830704 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:14:44.830789 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:44.831082 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:44.831088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:44.831093 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:14:44.831295 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 02:14:44.831509 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":280,"threads_started":1}
I20260926 02:14:44.833451 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 6968
I20260926 02:14:44.833609 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:14:44.833973 7096 heartbeater.cc:347] Connected to a master server at 127.28.152.190:41439
I20260926 02:14:44.834097 29282 external_mini_cluster.cc:1428] Setting key 07232d2c37781fa81837b67226ffc45e
I20260926 02:14:44.834270 7096 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:44.834920 7096 heartbeater.cc:511] Master 127.28.152.190:41439 requested a full tablet report, sending...
I20260926 02:14:44.836342 6644 ts_manager.cc:194] Registered new tserver with Master: e946f757561c4d1c8938ae6c6ae7ed86 (127.28.152.131:34221)
I20260926 02:14:44.849363 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 02:14:44.855400 6834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:14:44.853840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.28.152.190:41439)
0926 02:14:44.854015 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:44.854020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:44.854481 (+ 461us) server_negotiation.cc:407] Connection header received
0926 02:14:44.854655 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:44.854659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:44.854714 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:44.854793 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:44.855114 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49266: BlockingRecv error: recv got EOF from 127.0.0.1:49266 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:14:44.855757 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 6704
I20260926 02:14:44.859270 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:44.856806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.28.152.129:38375)
0926 02:14:44.857333 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:44.857339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:44.857367 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:14:44.857424 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:44.857430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:44.857644 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:44.857766 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:44.858630 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:44.858637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:44.858649 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:44.858666 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:44.858704 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:44.858717 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:44.858821 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:44.858876 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 02:14:44.858987 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":246,"threads_started":1}
I20260926 02:14:44.860484 6827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:44.941676 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 6704
I20260926 02:14:44.955113 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 6836
I20260926 02:14:44.958845 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:44.956120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59994 (local address 127.28.152.130:46401)
0926 02:14:44.956618 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:44.956625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:44.956652 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:14:44.956713 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:44.956719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:44.956948 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:44.957100 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:44.958128 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:44.958135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:44.958148 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:44.958166 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:44.958205 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:44.958219 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:44.958340 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:44.958482 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 02:14:44.958585 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":242,"threads_started":1}
I20260926 02:14:44.960215 6960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:45.039889 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 6836
I20260926 02:14:45.053030 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 6968
I20260926 02:14:45.056916 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:45.054015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.28.152.131:34221)
0926 02:14:45.054602 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:45.054609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:45.054638 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:14:45.054697 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:45.054704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:45.054913 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:45.055057 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:45.056179 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:45.056185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:45.056198 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:14:45.056214 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:45.056251 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:14:45.056263 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:45.056418 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:45.056527 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 02:14:45.056626 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":285,"threads_started":1}
I20260926 02:14:45.058145 7091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:45.137384 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 6968
I20260926 02:14:45.150080 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 6614
I20260926 02:14:45.151232 6675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:45.250634 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 6614
2026-09-26T02:14:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2336 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:14:45 dist-test-slave-hllj krb5kdc[7114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:14:45 dist-test-slave-hllj krb5kdc[7114](info): set up 2 sockets
Sep 26 02:14:45 dist-test-slave-hllj krb5kdc[7114](info): commencing operation
krb5kdc: starting...
W20260926 02:14:46.263119 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:14:46 dist-test-slave-hllj krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:14:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:14:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:14:46.496789 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:39977
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:39977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:46.824044 7127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:46.824371 7127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:46.824479 7127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:46.831879 7127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:14:46.832008 7127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:46.832065 7127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:46.832146 7127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:14:46.832208 7127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:14:46.842190 7127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:39977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:39977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:46.843565 7127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:46.844900 7127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:46.856808 7132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:46.856771 7135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:46.856837 7133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:46.857662 7127 server_base.cc:1034] running on GCE node
I20260926 02:14:46.858551 7127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:46.860190 7127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:46.861428 7127 hybrid_clock.cc:648] HybridClock initialized: now 1790388886861358 us; error 50 us; skew 500 ppm
I20260926 02:14:46.861828 7127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:46 dist-test-slave-hllj krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388886, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:46.865033 7127 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:14:46.866720 7127 webserver.cc:460] Webserver started at http://127.28.152.190:43371/ using document root <none> and password file <none>
I20260926 02:14:46.867524 7127 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:46.867656 7127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:46.867964 7127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:46.871255 7127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "9f521d6465ac4467b734ad56a03a3419"
format_stamp: "Formatted at 2026-09-26 02:14:46 on dist-test-slave-hllj"
server_key: "d2a2f80342ce97173e85d4272f064cc2"
server_key_iv: "50830139560431d7b13fbf1753ecadb6"
server_key_version: "encryptionkey@0"
I20260926 02:14:46.872128 7127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "9f521d6465ac4467b734ad56a03a3419"
format_stamp: "Formatted at 2026-09-26 02:14:46 on dist-test-slave-hllj"
server_key: "d2a2f80342ce97173e85d4272f064cc2"
server_key_iv: "50830139560431d7b13fbf1753ecadb6"
server_key_version: "encryptionkey@0"
I20260926 02:14:46.877918 7127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:14:46.881796 7142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:46.883476 7127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:14:46.883680 7127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "9f521d6465ac4467b734ad56a03a3419"
format_stamp: "Formatted at 2026-09-26 02:14:46 on dist-test-slave-hllj"
server_key: "d2a2f80342ce97173e85d4272f064cc2"
server_key_iv: "50830139560431d7b13fbf1753ecadb6"
server_key_version: "encryptionkey@0"
I20260926 02:14:46.883900 7127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:46.904904 7127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:46.910116 7127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:46.910454 7127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:46.930455 7127 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:39977
I20260926 02:14:46.930469 7194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:39977 every 8 connection(s)
I20260926 02:14:46.931988 7127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:14:46.936270 7195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:46.939145 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 7127
I20260926 02:14:46.939358 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:14:46.940151 29282 external_mini_cluster.cc:1428] Setting key f888d22968e4bd3d14affe0d052c66e8
I20260926 02:14:46.949771 7195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419: Bootstrap starting.
I20260926 02:14:46.953276 7195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:46.954432 7195 log.cc:824] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:14:46 dist-test-slave-hllj krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:46.957980 7195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419: No bootstrap required, opened a new log
I20260926 02:14:46.965015 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:46.942647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.28.152.190:39977)
0926 02:14:46.943607 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:46.943621 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:46.943670 (+ 49us) server_negotiation.cc:407] Connection header received
0926 02:14:46.945737 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:46.945772 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:46.946330 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:46.946889 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:46.948550 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:46.950137 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:46.951594 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:46.952369 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:46.952563 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:46.956912 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:46.956921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:46.956941 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:46.956984 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:46.960323 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:46.961252 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:46.961260 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:46.961268 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:46.961476 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:46.961906 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:46.961910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:46.961913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:46.962555 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:46.962881 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:46.963382 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 02:14:46.963851 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":432,"threads_started":1}
I20260926 02:14:46.966374 7195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f521d6465ac4467b734ad56a03a3419" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39977 } }
I20260926 02:14:46.966877 7195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:46.967000 7195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f521d6465ac4467b734ad56a03a3419, State: Initialized, Role: FOLLOWER
I20260926 02:14:46.967710 7195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f521d6465ac4467b734ad56a03a3419" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39977 } }
I20260926 02:14:46.967995 7195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:14:46.968148 7195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:14:46.968323 7195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:46.970577 7195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f521d6465ac4467b734ad56a03a3419" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39977 } }
I20260926 02:14:46.971082 7195 leader_election.cc:302] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f521d6465ac4467b734ad56a03a3419; no voters:
I20260926 02:14:46.971643 7195 leader_election.cc:288] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:14:46.971776 7200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:46.972390 7200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [term 1 LEADER]: Becoming Leader. State: Replica: 9f521d6465ac4467b734ad56a03a3419, State: Running, Role: LEADER
I20260926 02:14:46.972925 7200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f521d6465ac4467b734ad56a03a3419" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39977 } }
I20260926 02:14:46.973651 7195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:14:46.975860 7202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f521d6465ac4467b734ad56a03a3419. Latest consensus state: current_term: 1 leader_uuid: "9f521d6465ac4467b734ad56a03a3419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f521d6465ac4467b734ad56a03a3419" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39977 } } }
I20260926 02:14:46.976225 7202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:46.977603 7206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:14:46.977555 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f521d6465ac4467b734ad56a03a3419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f521d6465ac4467b734ad56a03a3419" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39977 } } }
I20260926 02:14:46.977885 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419 [sys.catalog]: This master's current role is: LEADER
I20260926 02:14:46.983661 7206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:14:46.994071 7206 catalog_manager.cc:1347] Generated new cluster ID: abfe5e07a12f493eab4ce0e1c39da1a7
I20260926 02:14:46.994189 7206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:14:47.006963 7206 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:14:47.008332 7206 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:14:47.015583 7206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419: Generated new TSK 0
I20260926 02:14:47.016465 7206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:14:47.076866 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39977
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:47.405290 7221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:47.405588 7221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:47.405683 7221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:47.413022 7221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:47.413173 7221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:47.413316 7221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:14:47.423373 7221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:47.424640 7221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:47.426021 7221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:47.439610 7226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:47.439641 7229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:47.439584 7227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:47.440668 7221 server_base.cc:1034] running on GCE node
I20260926 02:14:47.441341 7221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:47.442313 7221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:47.443517 7221 hybrid_clock.cc:648] HybridClock initialized: now 1790388887443458 us; error 59 us; skew 500 ppm
I20260926 02:14:47.443837 7221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:47 dist-test-slave-hllj krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388887, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:47.447109 7221 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:14:47.448717 7221 webserver.cc:460] Webserver started at http://127.28.152.129:37561/ using document root <none> and password file <none>
I20260926 02:14:47.449493 7221 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:47.449618 7221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:47.449918 7221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:47.453104 7221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "893a64ffe7d847a29be8191ef0d5fa28"
format_stamp: "Formatted at 2026-09-26 02:14:47 on dist-test-slave-hllj"
server_key: "7866496ab5ad0d0ef2722397bfa43c45"
server_key_iv: "ffe9b3f8351b193e05a3e4b852328b12"
server_key_version: "encryptionkey@0"
I20260926 02:14:47.453953 7221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "893a64ffe7d847a29be8191ef0d5fa28"
format_stamp: "Formatted at 2026-09-26 02:14:47 on dist-test-slave-hllj"
server_key: "7866496ab5ad0d0ef2722397bfa43c45"
server_key_iv: "ffe9b3f8351b193e05a3e4b852328b12"
server_key_version: "encryptionkey@0"
I20260926 02:14:47.459359 7221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 02:14:47.463027 7236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:47.464512 7221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:14:47.464676 7221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "893a64ffe7d847a29be8191ef0d5fa28"
format_stamp: "Formatted at 2026-09-26 02:14:47 on dist-test-slave-hllj"
server_key: "7866496ab5ad0d0ef2722397bfa43c45"
server_key_iv: "ffe9b3f8351b193e05a3e4b852328b12"
server_key_version: "encryptionkey@0"
I20260926 02:14:47.464867 7221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:47.506692 7221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:47.510954 7221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:47.511258 7221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:47.512295 7221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:47.513760 7221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:47.513842 7221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:47.513975 7221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:47.514029 7221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:47.544401 7221 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:45249
I20260926 02:14:47.544440 7348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:45249 every 8 connection(s)
I20260926 02:14:47.545761 7221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:14:47.549934 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 7221
I20260926 02:14:47.550133 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:14:47.550792 29282 external_mini_cluster.cc:1428] Setting key 524c63409f872724d85809bd958e166f
Sep 26 02:14:47 dist-test-slave-hllj krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388887, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:47.566629 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:47.548372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:49723 (local address 127.28.152.190:39977)
0926 02:14:47.548796 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:47.548801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:47.549889 (+ 1088us) server_negotiation.cc:407] Connection header received
0926 02:14:47.551904 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:47.551909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:47.551980 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:47.552084 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:47.553947 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:47.554660 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:47.556324 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:47.556746 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:47.556875 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:47.560871 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:47.560877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:47.560881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:47.560918 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:47.563404 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:47.564302 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:47.564306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:47.564308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:47.564424 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:47.564879 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:47.564883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:47.564885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:47.565169 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:47.565278 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:47.566182 (+ 904us) server_negotiation.cc:299] Negotiation successful
0926 02:14:47.566357 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":316,"thread_start_us":225,"threads_started":1}
I20260926 02:14:47.568204 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:47.548859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39977 (local address 127.28.152.129:49723)
0926 02:14:47.549621 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:47.549660 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:47.551587 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:47.552384 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:47.552396 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:47.553110 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:47.553775 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:47.553794 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:47.554823 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:47.554827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:47.556151 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:47.556166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:47.556995 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:47.557002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:47.557315 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:47.558000 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:47.558025 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:47.560590 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:47.563578 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:47.563588 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:47.563610 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:47.564187 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:47.564513 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:47.564518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:47.564522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:47.564780 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:47.565295 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:47.565304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:47.565780 (+ 476us) client_negotiation.cc:769] Sending connection context
0926 02:14:47.566095 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 02:14:47.566637 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":287,"threads_started":1}
I20260926 02:14:47.569968 7349 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39977
I20260926 02:14:47.570349 7349 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:47.571336 7349 heartbeater.cc:511] Master 127.28.152.190:39977 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
I20260926 02:14:47.573841 7159 ts_manager.cc:194] Registered new tserver with Master: 893a64ffe7d847a29be8191ef0d5fa28 (127.28.152.129:45249)
I20260926 02:14:47.575392 7159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:49723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:14:47.604931 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39977
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:47.930611 7355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:47.930899 7355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:47.931039 7355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:47.938441 7355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:47.938573 7355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:47.938728 7355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:14:47.948614 7355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:47.950023 7355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:47.951395 7355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:47.965246 7362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:47.965216 7361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:47.965188 7364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:47.965962 7355 server_base.cc:1034] running on GCE node
I20260926 02:14:47.966598 7355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:47.967567 7355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:47.968765 7355 hybrid_clock.cc:648] HybridClock initialized: now 1790388887968708 us; error 48 us; skew 500 ppm
I20260926 02:14:47.969082 7355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:47 dist-test-slave-hllj krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388887, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:47.972183 7355 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:14:47.973690 7355 webserver.cc:460] Webserver started at http://127.28.152.130:33369/ using document root <none> and password file <none>
I20260926 02:14:47.974364 7355 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:47.974465 7355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:47.974762 7355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:47.977730 7355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "473a4b5d58d941a78854d1da4e4b8209"
format_stamp: "Formatted at 2026-09-26 02:14:47 on dist-test-slave-hllj"
server_key: "6229d1c159f0eab81fad58c107f5e513"
server_key_iv: "29ebe29d1635724a48b95c5b658c95bc"
server_key_version: "encryptionkey@0"
I20260926 02:14:47.978502 7355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "473a4b5d58d941a78854d1da4e4b8209"
format_stamp: "Formatted at 2026-09-26 02:14:47 on dist-test-slave-hllj"
server_key: "6229d1c159f0eab81fad58c107f5e513"
server_key_iv: "29ebe29d1635724a48b95c5b658c95bc"
server_key_version: "encryptionkey@0"
I20260926 02:14:47.983855 7355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:14:47.987390 7371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:47.988931 7355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:14:47.989073 7355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "473a4b5d58d941a78854d1da4e4b8209"
format_stamp: "Formatted at 2026-09-26 02:14:47 on dist-test-slave-hllj"
server_key: "6229d1c159f0eab81fad58c107f5e513"
server_key_iv: "29ebe29d1635724a48b95c5b658c95bc"
server_key_version: "encryptionkey@0"
I20260926 02:14:47.989301 7355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:48.011234 7355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:48.015760 7355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:48.016062 7355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:48.017086 7355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:48.018541 7355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:48.018646 7355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:48.018771 7355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:48.018821 7355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:48.047922 7355 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:41275
I20260926 02:14:48.047969 7484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:41275 every 8 connection(s)
I20260926 02:14:48.049223 7355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:14:48.057098 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 7355
I20260926 02:14:48.057319 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:14:48.057762 29282 external_mini_cluster.cc:1428] Setting key 4803fbeb73dac092358772eb2ddfcf39
Sep 26 02:14:48 dist-test-slave-hllj krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388887, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:48.070705 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.051748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:57029 (local address 127.28.152.190:39977)
0926 02:14:48.051942 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:48.051948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:48.053262 (+ 1314us) server_negotiation.cc:407] Connection header received
0926 02:14:48.055203 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:48.055207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:48.055279 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:48.055370 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:48.057304 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.058101 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.059957 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.060368 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.060467 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.064577 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:48.064582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:48.064586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:48.064622 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:48.067260 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:48.068234 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:48.068240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:48.068243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:48.068399 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:48.068897 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:48.068901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:48.068904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:48.069299 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:48.069438 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:48.070177 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 02:14:48.070393 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:14:48.071601 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.052308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39977 (local address 127.28.152.130:57029)
0926 02:14:48.053068 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:48.053115 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:48.054898 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:48.055598 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:48.055607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:48.056287 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:48.057054 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.057085 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.058270 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.058275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.059801 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.059816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.060592 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.060598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.060883 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.061618 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:48.061647 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:48.064312 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:48.067432 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:48.067442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:48.067464 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:48.068076 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:48.068520 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:48.068523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:48.068525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:48.068751 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:48.069408 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:48.069419 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:48.069795 (+ 376us) client_negotiation.cc:769] Sending connection context
0926 02:14:48.070040 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 02:14:48.070456 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":288,"threads_started":1}
I20260926 02:14:48.073086 7485 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39977
I20260926 02:14:48.073455 7485 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:48.074385 7485 heartbeater.cc:511] Master 127.28.152.190:39977 requested a full tablet report, sending...
I20260926 02:14:48.075995 7159 ts_manager.cc:194] Registered new tserver with Master: 473a4b5d58d941a78854d1da4e4b8209 (127.28.152.130:41275)
I20260926 02:14:48.076805 7159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:57029
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:14:48.113277 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39977
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:14:48.440488 7490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:14:48.440783 7490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:14:48.440905 7490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:14:48.448143 7490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:14:48.448261 7490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:14:48.448403 7490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:14:48.458418 7490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:14:48.459723 7490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:14:48.461030 7490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:14:48.474097 7496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:48.474118 7498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:14:48.474174 7495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:14:48.474524 7490 server_base.cc:1034] running on GCE node
I20260926 02:14:48.475370 7490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:14:48.476315 7490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:14:48.477564 7490 hybrid_clock.cc:648] HybridClock initialized: now 1790388888477484 us; error 56 us; skew 500 ppm
I20260926 02:14:48.477943 7490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:14:48 dist-test-slave-hllj krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388888, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:14:48.481280 7490 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:14:48.482788 7490 webserver.cc:460] Webserver started at http://127.28.152.131:45869/ using document root <none> and password file <none>
I20260926 02:14:48.483492 7490 fs_manager.cc:359] Metadata directory not provided
I20260926 02:14:48.483603 7490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:14:48.483932 7490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:14:48.487228 7490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "34d04a9883974265bb22248d97ff7054"
format_stamp: "Formatted at 2026-09-26 02:14:48 on dist-test-slave-hllj"
server_key: "ceb8d13f0f95adf6df8ee3a0e88f101d"
server_key_iv: "61981b54f1dad462d4582ce8df28d592"
server_key_version: "encryptionkey@0"
I20260926 02:14:48.488117 7490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "34d04a9883974265bb22248d97ff7054"
format_stamp: "Formatted at 2026-09-26 02:14:48 on dist-test-slave-hllj"
server_key: "ceb8d13f0f95adf6df8ee3a0e88f101d"
server_key_iv: "61981b54f1dad462d4582ce8df28d592"
server_key_version: "encryptionkey@0"
I20260926 02:14:48.493733 7490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 02:14:48.497815 7505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:48.499528 7490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:14:48.499706 7490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "34d04a9883974265bb22248d97ff7054"
format_stamp: "Formatted at 2026-09-26 02:14:48 on dist-test-slave-hllj"
server_key: "ceb8d13f0f95adf6df8ee3a0e88f101d"
server_key_iv: "61981b54f1dad462d4582ce8df28d592"
server_key_version: "encryptionkey@0"
I20260926 02:14:48.499931 7490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:14:48.514905 7490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:14:48.519305 7490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:14:48.519569 7490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:14:48.520470 7490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:14:48.521905 7490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:14:48.521983 7490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:48.522084 7490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:14:48.522141 7490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:14:48.551431 7490 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:40871
I20260926 02:14:48.551450 7618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:40871 every 8 connection(s)
I20260926 02:14:48.552711 7490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:14:48.555377 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 7490
I20260926 02:14:48.555564 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:14:48.555994 29282 external_mini_cluster.cc:1428] Setting key e492fb1525bf87dcf5a4c98ac2a53a37
Sep 26 02:14:48 dist-test-slave-hllj krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388888, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:14:48.573669 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.555456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:55251 (local address 127.28.152.190:39977)
0926 02:14:48.555651 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:48.555658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:48.557091 (+ 1433us) server_negotiation.cc:407] Connection header received
0926 02:14:48.559373 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:48.559378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:48.559470 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:48.559552 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:48.561355 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.562123 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.563744 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.564153 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.564250 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.567819 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:48.567824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:48.567827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:48.567862 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:48.570368 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:48.571307 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:48.571311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:48.571314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:48.571477 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:48.571977 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:48.571980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:48.571982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:48.572351 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:48.572465 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:48.573207 (+ 742us) server_negotiation.cc:299] Negotiation successful
0926 02:14:48.573409 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:14:48.574760 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.555953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39977 (local address 127.28.152.131:55251)
0926 02:14:48.556832 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:48.556952 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:48.559081 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:48.559826 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:48.559849 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:48.560466 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:14:48.561166 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.561195 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.562279 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.562283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.563604 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.563617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.564375 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.564381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.564624 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.565316 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:14:48.565341 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:14:48.567579 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:14:48.570516 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:48.570525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:48.570557 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:48.571175 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:48.571597 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:14:48.571601 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:14:48.571603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:14:48.571862 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:14:48.572461 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:14:48.572467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:14:48.572836 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 02:14:48.573065 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 02:14:48.573549 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":329,"threads_started":1}
I20260926 02:14:48.576050 7619 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39977
I20260926 02:14:48.576346 7619 heartbeater.cc:464] Registering TS with master...
I20260926 02:14:48.577085 7619 heartbeater.cc:511] Master 127.28.152.190:39977 requested a full tablet report, sending...
I20260926 02:14:48.578598 7159 ts_manager.cc:194] Registered new tserver with Master: 34d04a9883974265bb22248d97ff7054 (127.28.152.131:40871)
I20260926 02:14:48.579063 7349 heartbeater.cc:503] Master 127.28.152.190:39977 was elected leader, sending a full tablet report...
I20260926 02:14:48.579524 7159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:55251
I20260926 02:14:48.584846 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:14:48.603117 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.589912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.28.152.190:39977)
0926 02:14:48.590158 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:48.590165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:48.590537 (+ 372us) server_negotiation.cc:407] Connection header received
0926 02:14:48.590729 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:48.590733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:48.590796 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:48.590878 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:48.592608 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.593424 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.594703 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.595140 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.595254 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.597597 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:48.597602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:48.597606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:48.597643 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:48.600248 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:48.601010 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:48.601015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:48.601017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:48.601216 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:48.601613 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:48.601618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:48.601621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:48.601964 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:48.602218 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:48.602548 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 02:14:48.602778 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:14:48.607870 7158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:14:48.614902 7158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:14:48.651015 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.627636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.28.152.130:41275)
0926 02:14:48.640525 (+ 12889us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:48.640536 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:48.642661 (+ 2125us) server_negotiation.cc:407] Connection header received
0926 02:14:48.643101 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:48.643110 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:48.643401 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:48.643569 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:48.644191 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.645801 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.648504 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.649847 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.650019 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.650210 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:48.650521 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 02:14:48.650692 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12661,"thread_start_us":12903,"threads_started":1}
I20260926 02:14:48.651100 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.627334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:41275 (local address 127.0.0.1:45790)
0926 02:14:48.642535 (+ 15201us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:48.642555 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:48.642839 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:48.643735 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:48.643743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:48.643772 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:48.644026 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.644035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.645945 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.645951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.648350 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.648362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.650070 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.650076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.650262 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.650309 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 02:14:48.650443 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 02:14:48.650654 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15079,"mutex_wait_us":71,"thread_start_us":14152,"threads_started":1}
I20260926 02:14:48.651352 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.627521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.28.152.131:40871)
0926 02:14:48.641284 (+ 13763us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:48.641294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:48.642114 (+ 820us) server_negotiation.cc:407] Connection header received
0926 02:14:48.643732 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:48.643739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:48.643927 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:48.644085 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:48.644923 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.646457 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.649070 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.650042 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.650210 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.650378 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:48.650895 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 02:14:48.651059 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13577,"thread_start_us":13412,"threads_started":1}
I20260926 02:14:48.652125 7632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.627243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:40871 (local address 127.0.0.1:34544)
0926 02:14:48.641912 (+ 14669us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:48.641961 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:48.643544 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:48.644365 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:48.644368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:48.644387 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:48.644752 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.644762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.646621 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.646627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.648936 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.648947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.650514 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.650520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.650688 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.650715 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 02:14:48.651698 (+ 983us) client_negotiation.cc:240] Negotiation successful
0926 02:14:48.651832 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14381,"thread_start_us":15020,"threads_started":1}
I20260926 02:14:48.652269 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.629526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.28.152.129:45249)
0926 02:14:48.643544 (+ 14018us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:48.643552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:48.643578 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:14:48.643684 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:48.643691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:48.643920 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:48.644109 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:48.644794 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.645870 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.648956 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.650382 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.650558 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.650704 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:48.651781 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0926 02:14:48.651946 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13874,"thread_start_us":13713,"threads_started":1}
I20260926 02:14:48.652825 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.629358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45249 (local address 127.0.0.1:40094)
0926 02:14:48.642818 (+ 13460us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:48.642839 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:48.642977 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:48.644118 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:48.644122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:48.644146 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:48.644565 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.644581 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.645985 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.645990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.648777 (+ 2787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.648791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.650852 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.650858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.651036 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.651069 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:14:48.651475 (+ 406us) client_negotiation.cc:240] Negotiation successful
0926 02:14:48.652456 (+ 981us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13299,"mutex_wait_us":152,"thread_start_us":13108,"threads_started":1}
I20260926 02:14:48.655489 7554 tablet_service.cc:1467] Processing CreateTablet for tablet 622d0339264c4ef38b5f605e8f5003e0 (DEFAULT_TABLE table=test-table [id=2621315a7fe24596b7aed75de97722c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:48.655522 7419 tablet_service.cc:1467] Processing CreateTablet for tablet 622d0339264c4ef38b5f605e8f5003e0 (DEFAULT_TABLE table=test-table [id=2621315a7fe24596b7aed75de97722c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:48.656857 7554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622d0339264c4ef38b5f605e8f5003e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:48.656512 7284 tablet_service.cc:1467] Processing CreateTablet for tablet 622d0339264c4ef38b5f605e8f5003e0 (DEFAULT_TABLE table=test-table [id=2621315a7fe24596b7aed75de97722c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:14:48.657265 7419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622d0339264c4ef38b5f605e8f5003e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:48.657801 7284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622d0339264c4ef38b5f605e8f5003e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:14:48.668684 7637 tablet_bootstrap.cc:492] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: Bootstrap starting.
I20260926 02:14:48.672822 7637 tablet_bootstrap.cc:654] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:48.673350 7639 tablet_bootstrap.cc:492] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: Bootstrap starting.
I20260926 02:14:48.674427 7637 log.cc:824] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:48.677366 7639 tablet_bootstrap.cc:654] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:48.677902 7638 tablet_bootstrap.cc:492] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: Bootstrap starting.
I20260926 02:14:48.678943 7639 log.cc:824] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:48.681838 7638 tablet_bootstrap.cc:654] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: Neither blocks nor log segments found. Creating new log.
I20260926 02:14:48.682163 7637 tablet_bootstrap.cc:492] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: No bootstrap required, opened a new log
I20260926 02:14:48.682471 7637 ts_tablet_manager.cc:1397] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260926 02:14:48.683168 7639 tablet_bootstrap.cc:492] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: No bootstrap required, opened a new log
I20260926 02:14:48.683199 7638 log.cc:824] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: Log is configured to *not* fsync() on all Append() calls
I20260926 02:14:48.683437 7639 ts_tablet_manager.cc:1397] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 02:14:48.686661 7638 tablet_bootstrap.cc:492] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: No bootstrap required, opened a new log
I20260926 02:14:48.687034 7638 ts_tablet_manager.cc:1397] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 02:14:48.691454 7639 raft_consensus.cc:348] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d04a9883974265bb22248d97ff7054" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40871 } } peers { permanent_uuid: "473a4b5d58d941a78854d1da4e4b8209" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41275 } } peers { permanent_uuid: "893a64ffe7d847a29be8191ef0d5fa28" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45249 } }
I20260926 02:14:48.691895 7639 raft_consensus.cc:374] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:48.691987 7639 raft_consensus.cc:729] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473a4b5d58d941a78854d1da4e4b8209, State: Initialized, Role: FOLLOWER
I20260926 02:14:48.692636 7639 consensus_queue.cc:260] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d04a9883974265bb22248d97ff7054" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40871 } } peers { permanent_uuid: "473a4b5d58d941a78854d1da4e4b8209" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41275 } } peers { permanent_uuid: "893a64ffe7d847a29be8191ef0d5fa28" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45249 } }
I20260926 02:14:48.692595 7637 raft_consensus.cc:348] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d04a9883974265bb22248d97ff7054" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40871 } } peers { permanent_uuid: "473a4b5d58d941a78854d1da4e4b8209" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41275 } } peers { permanent_uuid: "893a64ffe7d847a29be8191ef0d5fa28" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45249 } }
I20260926 02:14:48.693035 7637 raft_consensus.cc:374] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:48.693181 7637 raft_consensus.cc:729] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d04a9883974265bb22248d97ff7054, State: Initialized, Role: FOLLOWER
I20260926 02:14:48.693868 7485 heartbeater.cc:503] Master 127.28.152.190:39977 was elected leader, sending a full tablet report...
I20260926 02:14:48.693819 7637 consensus_queue.cc:260] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d04a9883974265bb22248d97ff7054" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40871 } } peers { permanent_uuid: "473a4b5d58d941a78854d1da4e4b8209" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41275 } } peers { permanent_uuid: "893a64ffe7d847a29be8191ef0d5fa28" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45249 } }
I20260926 02:14:48.694741 7639 ts_tablet_manager.cc:1428] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260926 02:14:48.695601 7619 heartbeater.cc:503] Master 127.28.152.190:39977 was elected leader, sending a full tablet report...
I20260926 02:14:48.696257 7637 ts_tablet_manager.cc:1428] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 02:14:48.698601 7638 raft_consensus.cc:348] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d04a9883974265bb22248d97ff7054" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40871 } } peers { permanent_uuid: "473a4b5d58d941a78854d1da4e4b8209" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41275 } } peers { permanent_uuid: "893a64ffe7d847a29be8191ef0d5fa28" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45249 } }
I20260926 02:14:48.699038 7638 raft_consensus.cc:374] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:14:48.699153 7638 raft_consensus.cc:729] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893a64ffe7d847a29be8191ef0d5fa28, State: Initialized, Role: FOLLOWER
I20260926 02:14:48.699811 7638 consensus_queue.cc:260] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d04a9883974265bb22248d97ff7054" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40871 } } peers { permanent_uuid: "473a4b5d58d941a78854d1da4e4b8209" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41275 } } peers { permanent_uuid: "893a64ffe7d847a29be8191ef0d5fa28" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45249 } }
I20260926 02:14:48.701468 7638 ts_tablet_manager.cc:1428] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 02:14:48.797461 7644 raft_consensus.cc:482] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:14:48.797749 7644 raft_consensus.cc:504] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d04a9883974265bb22248d97ff7054" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40871 } } peers { permanent_uuid: "473a4b5d58d941a78854d1da4e4b8209" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41275 } } peers { permanent_uuid: "893a64ffe7d847a29be8191ef0d5fa28" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45249 } }
W20260926 02:14:48.798254 7350 tablet.cc:2369] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:14:48.799777 7644 leader_election.cc:288] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 473a4b5d58d941a78854d1da4e4b8209 (127.28.152.130:41275), 893a64ffe7d847a29be8191ef0d5fa28 (127.28.152.129:45249)
W20260926 02:14:48.801088 7486 tablet.cc:2369] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:14:48.804342 7620 tablet.cc:2369] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:14:48.806223 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.800275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:42315 (local address 127.28.152.130:41275)
0926 02:14:48.800483 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:48.800490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:48.800731 (+ 241us) server_negotiation.cc:407] Connection header received
0926 02:14:48.800984 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:48.800989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:48.801071 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:48.801196 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:48.801624 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.802492 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.804234 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.805415 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.805529 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.805581 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:48.805944 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 02:14:48.806037 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:14:48.806313 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.800292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:41275 (local address 127.28.152.131:42315)
0926 02:14:48.800615 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:48.800630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:48.800784 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:48.801228 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:48.801233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:48.801266 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:48.801470 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.801482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.802630 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.802634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.804070 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.804083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.805560 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.805567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.805765 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.805802 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:14:48.805901 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 02:14:48.806039 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":20}
I20260926 02:14:48.807258 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.800389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45249 (local address 127.28.152.131:47295)
0926 02:14:48.801122 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 02:14:48.801174 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:14:48.801348 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:14:48.801719 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:14:48.801723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:14:48.801747 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:14:48.802131 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.802144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.803214 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.803218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.804993 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:14:48.805007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.806588 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.806595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:14:48.806757 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.806787 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:14:48.806897 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 02:14:48.807009 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":20,"thread_start_us":390,"threads_started":1}
I20260926 02:14:48.807365 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.800391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:47295 (local address 127.28.152.129:45249)
0926 02:14:48.800583 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:48.800588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:48.801243 (+ 655us) server_negotiation.cc:407] Connection header received
0926 02:14:48.801529 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:48.801533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:48.801605 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:48.801696 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:14:48.802298 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.803071 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.805226 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.806404 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.806557 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.806627 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:48.807009 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 02:14:48.807147 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:14:48.807502 7440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622d0339264c4ef38b5f605e8f5003e0" candidate_uuid: "34d04a9883974265bb22248d97ff7054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473a4b5d58d941a78854d1da4e4b8209" is_pre_election: true
I20260926 02:14:48.808051 7440 raft_consensus.cc:2454] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34d04a9883974265bb22248d97ff7054 in term 0.
I20260926 02:14:48.808230 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622d0339264c4ef38b5f605e8f5003e0" candidate_uuid: "34d04a9883974265bb22248d97ff7054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893a64ffe7d847a29be8191ef0d5fa28" is_pre_election: true
I20260926 02:14:48.808715 7304 raft_consensus.cc:2454] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34d04a9883974265bb22248d97ff7054 in term 0.
I20260926 02:14:48.808755 7508 leader_election.cc:302] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d04a9883974265bb22248d97ff7054, 473a4b5d58d941a78854d1da4e4b8209; no voters:
I20260926 02:14:48.809144 7644 raft_consensus.cc:2781] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:14:48.809302 7644 raft_consensus.cc:482] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:14:48.809412 7644 raft_consensus.cc:3037] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:48.811474 7644 raft_consensus.cc:504] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d04a9883974265bb22248d97ff7054" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40871 } } peers { permanent_uuid: "473a4b5d58d941a78854d1da4e4b8209" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41275 } } peers { permanent_uuid: "893a64ffe7d847a29be8191ef0d5fa28" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45249 } }
I20260926 02:14:48.812026 7644 leader_election.cc:288] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [CANDIDATE]: Term 1 election: Requested vote from peers 473a4b5d58d941a78854d1da4e4b8209 (127.28.152.130:41275), 893a64ffe7d847a29be8191ef0d5fa28 (127.28.152.129:45249)
I20260926 02:14:48.812532 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622d0339264c4ef38b5f605e8f5003e0" candidate_uuid: "34d04a9883974265bb22248d97ff7054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "893a64ffe7d847a29be8191ef0d5fa28"
I20260926 02:14:48.812549 7440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622d0339264c4ef38b5f605e8f5003e0" candidate_uuid: "34d04a9883974265bb22248d97ff7054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "473a4b5d58d941a78854d1da4e4b8209"
I20260926 02:14:48.812753 7440 raft_consensus.cc:3037] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:48.812754 7304 raft_consensus.cc:3037] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:14:48.816124 7440 raft_consensus.cc:2454] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d04a9883974265bb22248d97ff7054 in term 1.
I20260926 02:14:48.816196 7304 raft_consensus.cc:2454] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d04a9883974265bb22248d97ff7054 in term 1.
I20260926 02:14:48.816711 7507 leader_election.cc:302] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d04a9883974265bb22248d97ff7054, 893a64ffe7d847a29be8191ef0d5fa28; no voters:
I20260926 02:14:48.817097 7644 raft_consensus.cc:2781] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:14:48.817576 7644 raft_consensus.cc:686] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 1 LEADER]: Becoming Leader. State: Replica: 34d04a9883974265bb22248d97ff7054, State: Running, Role: LEADER
I20260926 02:14:48.818053 7644 consensus_queue.cc:237] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d04a9883974265bb22248d97ff7054" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40871 } } peers { permanent_uuid: "473a4b5d58d941a78854d1da4e4b8209" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41275 } } peers { permanent_uuid: "893a64ffe7d847a29be8191ef0d5fa28" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45249 } }
I20260926 02:14:48.822901 7159 catalog_manager.cc:5495] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34d04a9883974265bb22248d97ff7054 (127.28.152.131). New cstate: current_term: 1 leader_uuid: "34d04a9883974265bb22248d97ff7054" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d04a9883974265bb22248d97ff7054" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "473a4b5d58d941a78854d1da4e4b8209" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "893a64ffe7d847a29be8191ef0d5fa28" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45249 } health_report { overall_health: UNKNOWN } } }
I20260926 02:14:48.844559 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.839079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.28.152.131:40871)
0926 02:14:48.839287 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:48.839293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:48.839441 (+ 148us) server_negotiation.cc:407] Connection header received
0926 02:14:48.839660 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:48.839664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:48.839747 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:48.839880 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:14:48.840323 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.841054 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.842460 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.842851 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.842967 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.843469 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:14:48.843984 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:14:48.844130 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:48.844212 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 02:14:48.844298 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:14:48.853864 7440 raft_consensus.cc:1261] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209 [term 1 FOLLOWER]: Refusing update from remote peer 34d04a9883974265bb22248d97ff7054: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:48.853864 7304 raft_consensus.cc:1261] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28 [term 1 FOLLOWER]: Refusing update from remote peer 34d04a9883974265bb22248d97ff7054: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:14:48.854995 7647 consensus_queue.cc:1035] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "893a64ffe7d847a29be8191ef0d5fa28" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:48.855424 7644 consensus_queue.cc:1035] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "473a4b5d58d941a78854d1da4e4b8209" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:14:48.868805 7652 mvcc.cc:205] Tried to move back new op lower bound from 7333432888733581312 to 7333432888602415104. Current Snapshot: MvccSnapshot[applied={T|T < 7333432888733581312}]
I20260926 02:14:48.881160 7158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:48.881474 7158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:14:48.884868 7158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f521d6465ac4467b734ad56a03a3419: Sending DeleteTablet for 3 replicas of tablet 622d0339264c4ef38b5f605e8f5003e0
I20260926 02:14:48.886111 7554 tablet_service.cc:1514] Processing DeleteTablet for tablet 622d0339264c4ef38b5f605e8f5003e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:48 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:34544
I20260926 02:14:48.886214 7419 tablet_service.cc:1514] Processing DeleteTablet for tablet 622d0339264c4ef38b5f605e8f5003e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:48 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:45790
I20260926 02:14:48.886512 7284 tablet_service.cc:1514] Processing DeleteTablet for tablet 622d0339264c4ef38b5f605e8f5003e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:14:48 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:40094
I20260926 02:14:48.888756 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 7221
I20260926 02:14:48.901644 7662 tablet_replica.cc:330] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: stopping tablet replica
I20260926 02:14:48.902202 7662 raft_consensus.cc:2229] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:48.902822 7662 raft_consensus.cc:2258] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:48.905961 7662 ts_tablet_manager.cc:1905] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:48.909071 7660 tablet_replica.cc:330] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: stopping tablet replica
I20260926 02:14:48.909801 7660 raft_consensus.cc:2229] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:14:48.911633 7660 raft_consensus.cc:2258] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:14:48.914170 7662 ts_tablet_manager.cc:1918] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:48.914377 7662 log.cc:1196] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/622d0339264c4ef38b5f605e8f5003e0
I20260926 02:14:48.914943 7662 ts_tablet_manager.cc:1939] T 622d0339264c4ef38b5f605e8f5003e0 P 893a64ffe7d847a29be8191ef0d5fa28: Deleting consensus metadata
I20260926 02:14:48.917200 7660 ts_tablet_manager.cc:1905] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:48.917156 7144 catalog_manager.cc:4841] TS 893a64ffe7d847a29be8191ef0d5fa28 (127.28.152.129:45249): tablet 622d0339264c4ef38b5f605e8f5003e0 (table test-table [id=2621315a7fe24596b7aed75de97722c2]) successfully deleted
I20260926 02:14:48.917392 7661 tablet_replica.cc:330] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: stopping tablet replica
I20260926 02:14:48.917838 7661 raft_consensus.cc:2229] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:14:48.918509 7661 raft_consensus.cc:2258] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 02:14:48 dist-test-slave-hllj krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:14:48.920647 7661 ts_tablet_manager.cc:1905] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:14:48.924963 7660 ts_tablet_manager.cc:1918] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:48.925172 7660 log.cc:1196] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/622d0339264c4ef38b5f605e8f5003e0
I20260926 02:14:48.925721 7660 ts_tablet_manager.cc:1939] T 622d0339264c4ef38b5f605e8f5003e0 P 473a4b5d58d941a78854d1da4e4b8209: Deleting consensus metadata
I20260926 02:14:48.927755 7145 catalog_manager.cc:4841] TS 473a4b5d58d941a78854d1da4e4b8209 (127.28.152.130:41275): tablet 622d0339264c4ef38b5f605e8f5003e0 (table test-table [id=2621315a7fe24596b7aed75de97722c2]) successfully deleted
I20260926 02:14:48.930367 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:48.889960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.28.152.129:45249)
0926 02:14:48.890131 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:48.890137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:48.896163 (+ 6026us) server_negotiation.cc:407] Connection header received
0926 02:14:48.896374 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:48.896380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:48.896484 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:48.896602 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:48.904613 (+ 8011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.905812 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.910160 (+ 4348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:48.910864 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:48.910986 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:48.921428 (+ 10442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:48.921447 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:48.921467 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:48.921520 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:48.925859 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:48.926900 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:48.926912 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:48.926916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:48.927103 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:48.928316 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:48.928321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:48.928324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:48.928812 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:48.929206 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:48.929511 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 02:14:48.929766 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:14:48.931417 7661 ts_tablet_manager.cc:1918] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:14:48.931607 7661 log.cc:1196] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/622d0339264c4ef38b5f605e8f5003e0
I20260926 02:14:48.932114 7661 ts_tablet_manager.cc:1939] T 622d0339264c4ef38b5f605e8f5003e0 P 34d04a9883974265bb22248d97ff7054: Deleting consensus metadata
I20260926 02:14:48.932495 7344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:48.933895 7146 catalog_manager.cc:4841] TS 34d04a9883974265bb22248d97ff7054 (127.28.152.131:40871): tablet 622d0339264c4ef38b5f605e8f5003e0 (table test-table [id=2621315a7fe24596b7aed75de97722c2]) successfully deleted
I20260926 02:14:49.031600 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 7221
I20260926 02:14:49.052416 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 7355
Sep 26 02:14:49 dist-test-slave-hllj krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:14:49.072531 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:49.053366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.28.152.130:41275)
0926 02:14:49.053904 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:49.053909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:49.053938 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:14:49.053994 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:49.053999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:49.054073 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:49.054175 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:49.055729 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:49.056872 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:49.058937 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:49.059588 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:49.059708 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:49.065029 (+ 5321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:49.065044 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:49.065065 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:49.065121 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:49.069392 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:49.070277 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:49.070288 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:49.070292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:49.070451 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:49.070855 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:49.070860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:49.070864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:49.071358 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:49.071668 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:49.071915 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 02:14:49.072146 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424}
I20260926 02:14:49.073338 7480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:49.161911 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 7355
I20260926 02:14:49.183070 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 7490
Sep 26 02:14:49 dist-test-slave-hllj krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:14:49.198974 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:14:49.183793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.28.152.131:40871)
0926 02:14:49.183996 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:14:49.184001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:14:49.184021 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:14:49.184136 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:14:49.184142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:14:49.184204 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:14:49.184301 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:14:49.185587 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:49.186469 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:49.187810 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:14:49.188313 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:14:49.188428 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:14:49.192827 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:14:49.192844 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:14:49.192861 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:14:49.192904 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:14:49.195999 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:49.196762 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:49.196771 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:49.196776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:49.196941 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:14:49.197390 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:14:49.197394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:14:49.197396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:14:49.197785 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:14:49.198092 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 02:14:49.198393 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 02:14:49.198604 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:14:49.199769 7614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:49.320328 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 7490
I20260926 02:14:49.336601 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 7127
I20260926 02:14:49.337896 7190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:14:49.443986 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 7127
2026-09-26T02:14:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4197 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:14:49 dist-test-slave-hllj krb5kdc[7671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:14:49 dist-test-slave-hllj krb5kdc[7671](info): set up 2 sockets
Sep 26 02:14:49 dist-test-slave-hllj krb5kdc[7671](info): commencing operation
krb5kdc: starting...
W20260926 02:14:50.475378 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:14:50 dist-test-slave-hllj krb5kdc[7671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:14:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:14:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.152.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.152.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/rangeradmin_127.28.152.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/rangeradmin_127.28.152.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.152.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.152.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/rangerlookup_127.28.152.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/rangerlookup_127.28.152.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.152.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.152.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/HTTP_127.28.152.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/HTTP_127.28.152.148@KRBTEST.COM.keytab.
I20260926 02:14:50.803968 29282 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCd4s0O/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/postgres -l logfile start
2026-09-26 02:14:53.392 UTC [7701] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 02:14:53.392 UTC [7701] LOG: listening on IPv4 address "127.28.152.148", port 53509
2026-09-26 02:14:53.396 UTC [7701] LOG: listening on Unix socket "/tmp/.s.PGSQL.53509"
2026-09-26 02:14:53.418 UTC [7704] LOG: database system was shut down at 2026-09-26 02:14:51 UTC
2026-09-26 02:14:53.421 UTC [7701] LOG: database system is ready to accept connections
I20260926 02:14:54.311178 29282 mini_postgres.cc:96] Postgres bound to 53509
2026-09-26 02:14:54.330 UTC [7712] FATAL: database "slave" does not exist
127.28.152.148:53509 - accepting connections
I20260926 02:14:54.331552 29282 mini_ranger.cc:161] Starting Ranger...
I20260926 02:14:54.358156 29282 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 02:14:54.914793 29282 mini_ranger.cc:87] Created ranger Postgres database
I20260926 02:14:54.914958 29282 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-admin
2026-09-26 02:14:55,207 [I] DB FLAVOR :POSTGRES
2026-09-26 02:14:55,208 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:14:55,208 [I] Checking connection..
2026-09-26 02:14:55,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 02:14:55,448 [I] Checking connection passed.
2026-09-26 02:14:55,448 [I] --------- Verifying version history table ---------
2026-09-26 02:14:55,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:14:55,697 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:14:55,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:14:55,961 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:14:55,961 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 02:14:55,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 02:14:56.209 UTC [7776] WARNING: there is no transaction in progress
2026-09-26 02:14:56,215 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 02:14:56,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:14:56,475 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 02:14:56,476 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:14:56,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 02:14:56,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 02:14:57,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hllj.c.gcp-upstream.internal','N') ;"
2026-09-26 02:14:57,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:14:57,499 [I] Table x_portal_user does not exist in database ranger
2026-09-26 02:14:57,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:14:57,766 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 02:14:57,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:14:58,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:14:58,257 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 02:14:58,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 02:14:59.313 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.332 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.351 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.368 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.387 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.406 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.474 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.482 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.499 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.512 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.526 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.546 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.559 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.567 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.575 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:14:59.583 UTC [7885] WARNING: there is no transaction in progress
2026-09-26 02:15:00,307 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 02:15:00,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 02:15:00,561 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 02:15:00,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:15:00,800 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 02:15:00,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:15:01,065 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 02:15:01,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:15:01,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:15:01,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hllj.c.gcp-upstream.internal';"
2026-09-26 02:15:01,784 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 02:15:01,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 02:15:02,043 [I] DB_PATCHES have already been applied
I20260926 02:15:02.048928 29282 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCd4s0O/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 02:15:02.049047 29282 mini_ranger.cc:193] Using host: 127.28.152.148
I20260926 02:15:02.063390 29282 mini_ranger.cc:227] Ranger admin URL: http://127.28.152.148:32937
Sep 26, 2026 2:15:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:15:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:15:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:15:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:15:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:15:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:15:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 02:15:04.122476 29282 mini_ranger.cc:160] Time spent starting Ranger: real 9.791s user 0.002s sys 0.046s
Sep 26, 2026 2:15:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:15:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 2:15:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:15:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:15:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 02:15:25.233258 29282 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.152.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.152.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/rangerkms_127.28.152.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/rangerkms_127.28.152.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.152.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.152.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.152.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/HTTP_127.28.152.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.152.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/HTTP_127.28.152.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 02:15:25 dist-test-slave-hllj krb5kdc[7671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388925, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 02:15:25.417919 29282 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 02:15:25.445925 29282 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 02:15:26.435894 29282 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 02:15:26.436033 29282 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-kms
I20260926 02:15:26.436195 29282 mini_ranger_kms.cc:227] Using postgres at 127.28.152.148:53509
2026-09-26 02:15:26,843 [I] DB FLAVOR :POSTGRES
2026-09-26 02:15:26,843 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:15:26,843 [I] Checking connection
2026-09-26 02:15:26,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 02:15:27,076 [I] connection success
2026-09-26 02:15:27,077 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 02:15:27,077 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 02:15:27,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 02:15:27,312 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 02:15:27,312 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:15:27,312 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 02:15:27,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:53509/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 02:15:27,569 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 02:15:28.035244 29282 mini_ranger_kms.cc:310] Created kms service
I20260926 02:15:28.226962 29282 mini_ranger_kms.cc:326] Created kudu user
I20260926 02:15:28.278165 29282 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 02:15:28 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388925, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:15:28.690783 29282 mini_ranger_kms.cc:384] Added ranger policy
I20260926 02:15:28.691069 29282 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 02:15:28.691118 29282 mini_ranger_kms.cc:240] Using host: 127.28.152.148
I20260926 02:15:28.713150 29282 mini_ranger_kms.cc:276] Ranger KMS PID: 8068
I20260926 02:15:28.713339 29282 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.152.148:37121
Sep 26, 2026 2:15:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:15:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:15:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:15:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:15:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:15:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 02:15:30.843019 29282 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.425s user 0.097s sys 0.330s
I20260926 02:15:30.843274 29282 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 02:15:30.843466 29282 mini_ranger_kms.cc:401] 127.28.152.148:37121/kms/v1/keys
Sep 26 02:15:30 dist-test-slave-hllj krb5kdc[7671](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790388930, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.152.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 02:15:31 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388930, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.152.148@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
Sep 26 02:15:31 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388930, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.152.148@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
Sep 26 02:15:34 dist-test-slave-hllj krb5kdc[7671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:15:34.313220 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:46371
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:46371
--ranger_config_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:37121/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:15:34.664017 8119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:15:34.664340 8119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:15:34.664466 8119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:15:34.671790 8119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:15:34.671934 8119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:15:34.672013 8119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:15:34.672089 8119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:15:34.672169 8119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:15:34.683036 8119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.152.148:37121/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:46371
--ranger_config_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:46371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:15:34.684378 8119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:15:34.685765 8119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:15:34.698326 8124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:15:34.698218 8127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:15:34.698295 8125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:15:34.699429 8119 server_base.cc:1034] running on GCE node
I20260926 02:15:34.700302 8119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:15:34.701992 8119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:15:34.703202 8119 hybrid_clock.cc:648] HybridClock initialized: now 1790388934703160 us; error 41 us; skew 500 ppm
I20260926 02:15:34.703574 8119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:15:34 dist-test-slave-hllj krb5kdc[7671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388934, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:15:34.714212 8119 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:15:34.715902 8119 webserver.cc:460] Webserver started at http://127.28.152.190:35745/ using document root <none> and password file <none>
I20260926 02:15:34.716605 8119 fs_manager.cc:359] Metadata directory not provided
I20260926 02:15:34.716711 8119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:15:34.717023 8119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:15:34 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388934, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:15:34.785804 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "9546fa39ba644e71b467bc0933a2507e"
format_stamp: "Formatted at 2026-09-26 02:15:34 on dist-test-slave-hllj"
server_key: "7a0f13b1e08ea113fcd4d055baeb4325"
server_key_iv: "c3f79f246d893b384fec735fd67db6fb"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:34.786674 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "9546fa39ba644e71b467bc0933a2507e"
format_stamp: "Formatted at 2026-09-26 02:15:34 on dist-test-slave-hllj"
server_key: "7a0f13b1e08ea113fcd4d055baeb4325"
server_key_iv: "c3f79f246d893b384fec735fd67db6fb"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:34.792395 8119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:15:34.812788 8134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:15:34.814381 8119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:15:34.814555 8119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "9546fa39ba644e71b467bc0933a2507e"
format_stamp: "Formatted at 2026-09-26 02:15:34 on dist-test-slave-hllj"
server_key: "7a0f13b1e08ea113fcd4d055baeb4325"
server_key_iv: "c3f79f246d893b384fec735fd67db6fb"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:34.814803 8119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:15:34.862257 8119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:15:34.866528 8119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:15:34.866811 8119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:15:34.887354 8119 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:46371
I20260926 02:15:34.887404 8186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:46371 every 8 connection(s)
I20260926 02:15:34.889056 8119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:15:34.893443 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:15:34.897153 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 8119
I20260926 02:15:34.897372 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:15:34.898162 29282 external_mini_cluster.cc:1428] Setting key 5025399bcaa48b39d6fefa7f90c1690f
I20260926 02:15:34.908528 8187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e: Bootstrap starting.
I20260926 02:15:34.913734 8187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e: Neither blocks nor log segments found. Creating new log.
I20260926 02:15:34.915544 8187 log.cc:824] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:15:34.919064 8187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e: No bootstrap required, opened a new log
I20260926 02:15:34.928181 8187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9546fa39ba644e71b467bc0933a2507e" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46371 } }
I20260926 02:15:34.928689 8187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:15:34.928843 8187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9546fa39ba644e71b467bc0933a2507e, State: Initialized, Role: FOLLOWER
I20260926 02:15:34.929628 8187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9546fa39ba644e71b467bc0933a2507e" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46371 } }
I20260926 02:15:34.929939 8187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:15:34.930115 8187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:15:34.930296 8187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [term 0 FOLLOWER]: Advancing to term 1
Sep 26 02:15:34 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:15:34.932806 8187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9546fa39ba644e71b467bc0933a2507e" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46371 } }
I20260926 02:15:34.933462 8187 leader_election.cc:302] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9546fa39ba644e71b467bc0933a2507e; no voters:
I20260926 02:15:34.934229 8187 leader_election.cc:288] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:15:34.934656 8192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:15:34.941658 8192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [term 1 LEADER]: Becoming Leader. State: Replica: 9546fa39ba644e71b467bc0933a2507e, State: Running, Role: LEADER
I20260926 02:15:34.942682 8192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9546fa39ba644e71b467bc0933a2507e" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46371 } }
I20260926 02:15:34.943526 8187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:15:34.943562 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:34.900378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.28.152.190:46371)
0926 02:15:34.915448 (+ 15070us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:34.915467 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:34.915536 (+ 69us) server_negotiation.cc:407] Connection header received
0926 02:15:34.918223 (+ 2687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:34.918262 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:34.918869 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:34.920991 (+ 2122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:15:34.921661 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:34.923786 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:34.926084 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:34.927269 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:34.927466 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:34.932341 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:15:34.932353 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:15:34.932376 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:15:34.932436 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:15:34.936668 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:34.937846 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:34.937857 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:34.937869 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:34.938109 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:34.938607 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:34.938612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:34.938616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:34.939462 (+ 846us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:15:34.940968 (+ 1506us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:34.941319 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 02:15:34.941776 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14799,"thread_start_us":14561,"threads_started":1}
I20260926 02:15:34.953755 8193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9546fa39ba644e71b467bc0933a2507e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9546fa39ba644e71b467bc0933a2507e" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46371 } } }
I20260926 02:15:34.954146 8193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [sys.catalog]: This master's current role is: LEADER
I20260926 02:15:34.954741 8194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9546fa39ba644e71b467bc0933a2507e. Latest consensus state: current_term: 1 leader_uuid: "9546fa39ba644e71b467bc0933a2507e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9546fa39ba644e71b467bc0933a2507e" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46371 } } }
I20260926 02:15:34.955041 8194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e [sys.catalog]: This master's current role is: LEADER
I20260926 02:15:34.958484 8187 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 02:15:34.966315 8206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:15:34.982590 8206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:15:35.010442 8206 catalog_manager.cc:1347] Generated new cluster ID: 5f1442fe7ece45aebc88e040458c7dcf
I20260926 02:15:35.010567 8206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:15:35.032363 8206 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:15:35.034579 8206 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:15:35.044890 8206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e: Generated new TSK 0
I20260926 02:15:35.046254 8206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:15:35.797464 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:37121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46371
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:15:36 dist-test-slave-hllj krb5kdc[7671](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790388936, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 02:15:36.191502 8232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:15:36.191805 8232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:15:36.191975 8232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:15:36.200860 8232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:15:36.201090 8232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:15:36.201282 8232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:15:36.212095 8232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.152.148:37121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:15:36.213995 8232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:15:36.215636 8232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:15:36.230156 8242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:15:36.230561 8243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:15:36.230919 8245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:15:36.231235 8232 server_base.cc:1034] running on GCE node
I20260926 02:15:36.231891 8232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:15:36.233017 8232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:15:36.234315 8232 hybrid_clock.cc:648] HybridClock initialized: now 1790388936234256 us; error 51 us; skew 500 ppm
I20260926 02:15:36.234658 8232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:15:36 dist-test-slave-hllj krb5kdc[7671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388936, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:15:36.238320 8232 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:15:36.242455 8232 webserver.cc:460] Webserver started at http://127.28.152.129:44943/ using document root <none> and password file <none>
I20260926 02:15:36.243196 8232 fs_manager.cc:359] Metadata directory not provided
I20260926 02:15:36.243422 8232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:15:36.243847 8232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:15:36 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388936, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:15:36.266739 8232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "10da590321554ba8bc43fe38004fa46c"
format_stamp: "Formatted at 2026-09-26 02:15:36 on dist-test-slave-hllj"
server_key: "c0860b856ed2fac34ccf7686d237f132"
server_key_iv: "cf85250f0db56ea9f3cf9daa26e3e8e1"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:36.267627 8232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "10da590321554ba8bc43fe38004fa46c"
format_stamp: "Formatted at 2026-09-26 02:15:36 on dist-test-slave-hllj"
server_key: "c0860b856ed2fac34ccf7686d237f132"
server_key_iv: "cf85250f0db56ea9f3cf9daa26e3e8e1"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:36.278924 8232 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.010s sys 0.003s
I20260926 02:15:36.307340 8252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:15:36.309182 8232 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.004s
I20260926 02:15:36.309398 8232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "10da590321554ba8bc43fe38004fa46c"
format_stamp: "Formatted at 2026-09-26 02:15:36 on dist-test-slave-hllj"
server_key: "c0860b856ed2fac34ccf7686d237f132"
server_key_iv: "cf85250f0db56ea9f3cf9daa26e3e8e1"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:36.309629 8232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:15:36.337000 8232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:15:36.341687 8232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:15:36.342406 8232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:15:36.343454 8232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:15:36.351840 8232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:15:36.351976 8232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:15:36.352144 8232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:15:36.352243 8232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:15:36.427542 8232 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:45803
I20260926 02:15:36.427750 8365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:45803 every 8 connection(s)
I20260926 02:15:36.429558 8232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:15:36.441706 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 8232
I20260926 02:15:36.441936 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:15:36.442740 29282 external_mini_cluster.cc:1428] Setting key eaac21af44f8d0e966e55cacf81ddb18
Sep 26 02:15:36 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388936, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
I20260926 02:15:36.491797 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:36.432770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:56467 (local address 127.28.152.190:46371)
0926 02:15:36.470542 (+ 37772us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:36.470549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:36.470579 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:15:36.471175 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:36.471182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:36.471311 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:36.471423 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:15:36.473929 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:36.475104 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:36.477733 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:36.478760 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:36.478873 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:36.483953 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:15:36.483962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:15:36.483968 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:15:36.484014 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:15:36.487576 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:36.488692 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:36.488699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:36.488703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:36.488867 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:36.489534 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:36.489540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:36.489544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:36.489915 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:15:36.490068 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:36.491092 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0926 02:15:36.491351 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37620,"thread_start_us":37497,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
I20260926 02:15:36.493072 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:36.433629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:46371 (local address 127.28.152.129:56467)
0926 02:15:36.466476 (+ 32847us) negotiation.cc:107] Waiting for socket to connect
0926 02:15:36.466551 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 02:15:36.470611 (+ 4060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:15:36.471935 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:15:36.471949 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:15:36.472649 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:15:36.473669 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:36.473703 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:36.475346 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:36.475352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:36.477523 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:36.477552 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:36.479263 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:36.479271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:36.479637 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:36.480748 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:15:36.480784 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:15:36.483625 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:15:36.487792 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:15:36.487805 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:15:36.487840 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:15:36.488535 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:15:36.489008 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:15:36.489014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:15:36.489017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:15:36.489352 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:15:36.490063 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:15:36.490073 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:15:36.490604 (+ 531us) client_negotiation.cc:769] Sending connection context
0926 02:15:36.490947 (+ 343us) client_negotiation.cc:240] Negotiation successful
0926 02:15:36.491564 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32412,"thread_start_us":35374,"threads_started":1}
I20260926 02:15:36.502619 8366 heartbeater.cc:347] Connected to a master server at 127.28.152.190:46371
I20260926 02:15:36.503060 8366 heartbeater.cc:464] Registering TS with master...
I20260926 02:15:36.504086 8366 heartbeater.cc:511] Master 127.28.152.190:46371 requested a full tablet report, sending...
I20260926 02:15:36.506801 8152 ts_manager.cc:194] Registered new tserver with Master: 10da590321554ba8bc43fe38004fa46c (127.28.152.129:45803)
I20260926 02:15:36.508845 8152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:56467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:15:36.528640 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:37121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46371
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:15:36 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388936, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
Sep 26 02:15:36 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388936, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
W20260926 02:15:37.220299 8372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:15:37.220762 8372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:15:37.220980 8372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:15:37.232723 8372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:15:37.232957 8372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:15:37.233198 8372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:15:37.249320 8372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.152.148:37121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:15:37.251349 8372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:15:37.253230 8372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:15:37.268504 8389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:15:37.268900 8387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:15:37.268461 8386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:15:37.270644 8372 server_base.cc:1034] running on GCE node
I20260926 02:15:37.271256 8372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:15:37.272321 8372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:15:37.273566 8372 hybrid_clock.cc:648] HybridClock initialized: now 1790388937273494 us; error 63 us; skew 500 ppm
I20260926 02:15:37.273978 8372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:15:37 dist-test-slave-hllj krb5kdc[7671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388937, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:15:37.277541 8372 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:15:37.279347 8372 webserver.cc:460] Webserver started at http://127.28.152.130:33663/ using document root <none> and password file <none>
I20260926 02:15:37.280153 8372 fs_manager.cc:359] Metadata directory not provided
I20260926 02:15:37.280308 8372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:15:37.280664 8372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:15:37 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388937, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:15:37.300225 8372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "e3ac260b1f564180a7c66957a311e0ec"
format_stamp: "Formatted at 2026-09-26 02:15:37 on dist-test-slave-hllj"
server_key: "d34260e8afec6f0b3c32fc42cc0c1d00"
server_key_iv: "6277d8f7f29adef503f5af3641f7d611"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:37.301483 8372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "e3ac260b1f564180a7c66957a311e0ec"
format_stamp: "Formatted at 2026-09-26 02:15:37 on dist-test-slave-hllj"
server_key: "d34260e8afec6f0b3c32fc42cc0c1d00"
server_key_iv: "6277d8f7f29adef503f5af3641f7d611"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:37.309264 8372 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 02:15:37.335474 8397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:15:37.337476 8372 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.007s sys 0.003s
I20260926 02:15:37.337675 8372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "e3ac260b1f564180a7c66957a311e0ec"
format_stamp: "Formatted at 2026-09-26 02:15:37 on dist-test-slave-hllj"
server_key: "d34260e8afec6f0b3c32fc42cc0c1d00"
server_key_iv: "6277d8f7f29adef503f5af3641f7d611"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:37.337931 8372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:15:37.357877 8372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:15:37.362360 8372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:15:37.362673 8372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:15:37.363790 8372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:15:37.365288 8372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:15:37.365361 8372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:15:37.365476 8372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:15:37.365525 8372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:15:37.397818 8372 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:41979
I20260926 02:15:37.397823 8510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:41979 every 8 connection(s)
I20260926 02:15:37.399134 8372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:15:37.407274 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 8372
I20260926 02:15:37.407454 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:15:37.407999 29282 external_mini_cluster.cc:1428] Setting key f9684ac285c645211618d668e626372a
Sep 26 02:15:37 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388937, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:15:37.423986 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:37.401989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:37025 (local address 127.28.152.190:46371)
0926 02:15:37.402397 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:37.402401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:37.403642 (+ 1241us) server_negotiation.cc:407] Connection header received
0926 02:15:37.405533 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:37.405538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:37.405628 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:37.405723 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:15:37.407244 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:37.408127 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:37.410235 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:37.410856 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:37.410992 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:37.415251 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:15:37.415257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:15:37.415262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:15:37.415302 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:15:37.418420 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:37.419219 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:37.419225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:37.419229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:37.419366 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:37.421791 (+ 2425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:37.421797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:37.421801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:37.422179 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:15:37.422337 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:37.423307 (+ 970us) server_negotiation.cc:299] Negotiation successful
0926 02:15:37.423604 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":201,"threads_started":1}
I20260926 02:15:37.425287 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:37.402715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:46371 (local address 127.28.152.130:37025)
0926 02:15:37.403478 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 02:15:37.403516 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:15:37.405258 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:15:37.405991 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:15:37.406000 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:15:37.406446 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:15:37.407057 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:37.407075 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:37.408290 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:37.408293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:37.410086 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:37.410102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:37.411109 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:37.411115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:37.411397 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:37.412186 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:15:37.412215 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:15:37.414985 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:15:37.418588 (+ 3603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:15:37.418599 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:15:37.418619 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:15:37.419083 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:15:37.421286 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:15:37.421292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:15:37.421296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:15:37.421639 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:15:37.422329 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:15:37.422339 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:15:37.422831 (+ 492us) client_negotiation.cc:769] Sending connection context
0926 02:15:37.423172 (+ 341us) client_negotiation.cc:240] Negotiation successful
0926 02:15:37.423737 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":316,"threads_started":1}
I20260926 02:15:37.427372 8511 heartbeater.cc:347] Connected to a master server at 127.28.152.190:46371
I20260926 02:15:37.427742 8511 heartbeater.cc:464] Registering TS with master...
I20260926 02:15:37.428821 8511 heartbeater.cc:511] Master 127.28.152.190:46371 requested a full tablet report, sending...
I20260926 02:15:37.430935 8152 ts_manager.cc:194] Registered new tserver with Master: e3ac260b1f564180a7c66957a311e0ec (127.28.152.130:41979)
I20260926 02:15:37.431993 8152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:37025
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:15:37.465498 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:37121/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46371
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:15:37.512857 8366 heartbeater.cc:503] Master 127.28.152.190:46371 was elected leader, sending a full tablet report...
W20260926 02:15:37.807037 8517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:15:37.807356 8517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:15:37.807476 8517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:15:37.815280 8517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:15:37.815424 8517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:15:37.815580 8517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:15:37.825680 8517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.152.148:37121/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:15:37.827030 8517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:15:37.828365 8517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:15:37.842051 8525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:15:37.842109 8523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:15:37.842289 8522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:15:37.842810 8517 server_base.cc:1034] running on GCE node
I20260926 02:15:37.843453 8517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:15:37.844491 8517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:15:37.845687 8517 hybrid_clock.cc:648] HybridClock initialized: now 1790388937845630 us; error 44 us; skew 500 ppm
I20260926 02:15:37.845997 8517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:15:37 dist-test-slave-hllj krb5kdc[7671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388937, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:15:37.848964 8517 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:15:37.850467 8517 webserver.cc:460] Webserver started at http://127.28.152.131:43951/ using document root <none> and password file <none>
I20260926 02:15:37.851624 8517 fs_manager.cc:359] Metadata directory not provided
I20260926 02:15:37.851756 8517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:15:37.852063 8517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:15:37 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388937, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:15:37.866580 8517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "5c6f530130194d82912ce2380585680f"
format_stamp: "Formatted at 2026-09-26 02:15:37 on dist-test-slave-hllj"
server_key: "51aa5653e6047fce18da432330f1ee4c"
server_key_iv: "fb09cda7c4e2bba6282e570032f6ea38"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:37.867383 8517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "5c6f530130194d82912ce2380585680f"
format_stamp: "Formatted at 2026-09-26 02:15:37 on dist-test-slave-hllj"
server_key: "51aa5653e6047fce18da432330f1ee4c"
server_key_iv: "fb09cda7c4e2bba6282e570032f6ea38"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:37.873234 8517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:15:37.905472 8532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:15:37.907670 8517 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.012s
I20260926 02:15:37.907922 8517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "5c6f530130194d82912ce2380585680f"
format_stamp: "Formatted at 2026-09-26 02:15:37 on dist-test-slave-hllj"
server_key: "51aa5653e6047fce18da432330f1ee4c"
server_key_iv: "fb09cda7c4e2bba6282e570032f6ea38"
server_key_version: "kuduclusterkey@0"
I20260926 02:15:37.908218 8517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:15:37.930472 8517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:15:37.934559 8517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:15:37.934845 8517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:15:37.935833 8517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:15:37.937273 8517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:15:37.937353 8517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:15:37.937481 8517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:15:37.937532 8517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:15:37.974769 8517 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:41289
I20260926 02:15:37.975330 8645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:41289 every 8 connection(s)
I20260926 02:15:37.976182 8517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:15:37.978946 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 8517
I20260926 02:15:37.979127 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:15:37.979682 29282 external_mini_cluster.cc:1428] Setting key 7b807c79cc2e55e432f069091adbc466
Sep 26 02:15:37 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388937, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:15:37.997988 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:37.980400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:60027 (local address 127.28.152.190:46371)
0926 02:15:37.980883 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:37.980890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:37.980913 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:15:37.983138 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:37.983143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:37.983251 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:37.983344 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:15:37.985072 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:37.985829 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:37.987422 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:37.987843 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:37.987977 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:37.991194 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:15:37.991199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:15:37.991202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:15:37.991235 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:15:37.994352 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:37.995244 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:37.995251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:37.995253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:37.995409 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:37.996023 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:37.996029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:37.996032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:37.996376 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:15:37.996518 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:37.997482 (+ 964us) server_negotiation.cc:299] Negotiation successful
0926 02:15:37.997675 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":977,"threads_started":1}
I20260926 02:15:37.999444 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:37.979851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:46371 (local address 127.28.152.131:60027)
0926 02:15:37.980748 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 02:15:37.980807 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 02:15:37.982839 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:15:37.983627 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:15:37.983636 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:15:37.984195 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:15:37.984889 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:37.984910 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:37.985990 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:37.985995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:37.987271 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:37.987283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:37.988077 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:37.988084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:37.988318 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:37.988978 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:15:37.989001 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:15:37.990963 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:15:37.994544 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:15:37.994554 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:15:37.994569 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:15:37.995116 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:15:37.995539 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:15:37.995544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:15:37.995549 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:15:37.995878 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:15:37.996515 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:15:37.996525 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:15:37.997026 (+ 501us) client_negotiation.cc:769] Sending connection context
0926 02:15:37.997406 (+ 380us) client_negotiation.cc:240] Negotiation successful
0926 02:15:37.998002 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":515,"threads_started":1}
I20260926 02:15:38.000911 8646 heartbeater.cc:347] Connected to a master server at 127.28.152.190:46371
I20260926 02:15:38.001284 8646 heartbeater.cc:464] Registering TS with master...
I20260926 02:15:38.002146 8646 heartbeater.cc:511] Master 127.28.152.190:46371 requested a full tablet report, sending...
I20260926 02:15:38.003671 8152 ts_manager.cc:194] Registered new tserver with Master: 5c6f530130194d82912ce2380585680f (127.28.152.131:41289)
I20260926 02:15:38.004493 8152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:60027
I20260926 02:15:38.009256 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:15:38.027101 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.013776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.28.152.190:46371)
0926 02:15:38.013936 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:38.013941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:38.014396 (+ 455us) server_negotiation.cc:407] Connection header received
0926 02:15:38.014559 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:38.014563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:38.014643 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:38.014720 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:15:38.016119 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.016867 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.018456 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.018940 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.019071 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.021765 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:15:38.021771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:15:38.021774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:15:38.021810 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:15:38.024480 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:38.025230 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:38.025234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:38.025237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:38.025365 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:38.025711 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:38.025714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:38.025716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:38.025981 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:15:38.026578 (+ 597us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:38.026683 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 02:15:38.026855 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:15:38.031347 8152 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:15:38.038215 8152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:15:38.072667 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.050231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:41289 (local address 127.0.0.1:35418)
0926 02:15:38.058988 (+ 8757us) negotiation.cc:107] Waiting for socket to connect
0926 02:15:38.059055 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 02:15:38.059271 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:15:38.064272 (+ 5001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:15:38.064281 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:15:38.064324 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:15:38.064743 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:38.064755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.066804 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.066809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:38.069428 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:38.069440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.071334 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.071341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:38.071839 (+ 498us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.071897 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 02:15:38.072026 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 02:15:38.072233 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8414,"mutex_wait_us":247,"thread_start_us":9259,"threads_started":1}
I20260926 02:15:38.073289 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.050521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.28.152.129:45803)
0926 02:15:38.061264 (+ 10743us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:38.061274 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:38.065618 (+ 4344us) server_negotiation.cc:407] Connection header received
0926 02:15:38.065726 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:38.065734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:38.066033 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:38.066222 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:15:38.066816 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.068419 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.070513 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.071646 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.071832 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.072031 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:38.072775 (+ 744us) server_negotiation.cc:299] Negotiation successful
0926 02:15:38.072960 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10555,"thread_start_us":10364,"threads_started":1}
I20260926 02:15:38.073613 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.050530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.28.152.131:41289)
0926 02:15:38.063238 (+ 12708us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:38.063247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:38.063284 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:15:38.063394 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:38.063403 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:38.063698 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:38.063919 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:15:38.065196 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.066652 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.069612 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.071128 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.071316 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.071495 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:38.073067 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0926 02:15:38.073268 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12531,"thread_start_us":14355,"threads_started":1}
I20260926 02:15:38.073855 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.050464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45803 (local address 127.0.0.1:35072)
0926 02:15:38.065344 (+ 14880us) negotiation.cc:107] Waiting for socket to connect
0926 02:15:38.065364 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:15:38.065484 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:15:38.066275 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:15:38.066281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:15:38.066306 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:15:38.066647 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:38.066656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.068572 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.068577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:38.070362 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:38.070373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.072167 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.072173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:38.072345 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.072381 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:15:38.072462 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 02:15:38.072584 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14764,"mutex_wait_us":2,"thread_start_us":14602,"threads_started":1}
I20260926 02:15:38.074158 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.053256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:41979 (local address 127.0.0.1:32982)
0926 02:15:38.066055 (+ 12799us) negotiation.cc:107] Waiting for socket to connect
0926 02:15:38.066071 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:15:38.066190 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:15:38.066769 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:15:38.066773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:15:38.066794 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:15:38.067764 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:38.067776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.069657 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.069662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:38.071616 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:38.071626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.073260 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.073266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:38.073428 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.073457 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 02:15:38.073539 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 02:15:38.073673 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12690,"mutex_wait_us":28,"thread_start_us":12561,"threads_started":1}
I20260926 02:15:38.074383 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.053877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.28.152.130:41979)
0926 02:15:38.065685 (+ 11808us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:38.065692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:38.066181 (+ 489us) server_negotiation.cc:407] Connection header received
0926 02:15:38.066374 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:38.066393 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:38.066606 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:38.066754 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:15:38.067995 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.069521 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.071758 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.073048 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.073234 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.073402 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:38.073917 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 02:15:38.074090 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11646,"thread_start_us":11539,"threads_started":1}
I20260926 02:15:38.077612 8580 tablet_service.cc:1467] Processing CreateTablet for tablet 0de0c875558d4f92b9fba97e7f5a22f5 (DEFAULT_TABLE table=test-table [id=9eebf15f201d4e7c866e2f264c0a7fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:15:38.077956 8300 tablet_service.cc:1467] Processing CreateTablet for tablet 0de0c875558d4f92b9fba97e7f5a22f5 (DEFAULT_TABLE table=test-table [id=9eebf15f201d4e7c866e2f264c0a7fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:15:38.078035 8445 tablet_service.cc:1467] Processing CreateTablet for tablet 0de0c875558d4f92b9fba97e7f5a22f5 (DEFAULT_TABLE table=test-table [id=9eebf15f201d4e7c866e2f264c0a7fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:15:38.079064 8580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0de0c875558d4f92b9fba97e7f5a22f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:15:38.079268 8300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0de0c875558d4f92b9fba97e7f5a22f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:15:38.079284 8445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0de0c875558d4f92b9fba97e7f5a22f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:15:38.092167 8665 tablet_bootstrap.cc:492] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: Bootstrap starting.
I20260926 02:15:38.093847 8667 tablet_bootstrap.cc:492] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: Bootstrap starting.
I20260926 02:15:38.094630 8666 tablet_bootstrap.cc:492] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: Bootstrap starting.
I20260926 02:15:38.096228 8665 tablet_bootstrap.cc:654] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: Neither blocks nor log segments found. Creating new log.
I20260926 02:15:38.097599 8665 log.cc:824] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:15:38.098731 8666 tablet_bootstrap.cc:654] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: Neither blocks nor log segments found. Creating new log.
I20260926 02:15:38.099983 8666 log.cc:824] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: Log is configured to *not* fsync() on all Append() calls
I20260926 02:15:38.100729 8667 tablet_bootstrap.cc:654] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: Neither blocks nor log segments found. Creating new log.
I20260926 02:15:38.101665 8665 tablet_bootstrap.cc:492] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: No bootstrap required, opened a new log
I20260926 02:15:38.101943 8665 ts_tablet_manager.cc:1397] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260926 02:15:38.102262 8667 log.cc:824] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:15:38.104030 8666 tablet_bootstrap.cc:492] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: No bootstrap required, opened a new log
I20260926 02:15:38.104334 8666 ts_tablet_manager.cc:1397] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 02:15:38.104861 8667 tablet_bootstrap.cc:492] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: No bootstrap required, opened a new log
I20260926 02:15:38.105111 8667 ts_tablet_manager.cc:1397] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260926 02:15:38.112896 8665 raft_consensus.cc:348] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6f530130194d82912ce2380585680f" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41289 } } peers { permanent_uuid: "10da590321554ba8bc43fe38004fa46c" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45803 } } peers { permanent_uuid: "e3ac260b1f564180a7c66957a311e0ec" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41979 } }
I20260926 02:15:38.113404 8665 raft_consensus.cc:374] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:15:38.113204 8667 raft_consensus.cc:348] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6f530130194d82912ce2380585680f" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41289 } } peers { permanent_uuid: "10da590321554ba8bc43fe38004fa46c" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45803 } } peers { permanent_uuid: "e3ac260b1f564180a7c66957a311e0ec" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41979 } }
I20260926 02:15:38.113539 8665 raft_consensus.cc:729] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6f530130194d82912ce2380585680f, State: Initialized, Role: FOLLOWER
I20260926 02:15:38.113723 8667 raft_consensus.cc:374] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:15:38.113830 8667 raft_consensus.cc:729] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10da590321554ba8bc43fe38004fa46c, State: Initialized, Role: FOLLOWER
I20260926 02:15:38.114399 8665 consensus_queue.cc:260] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6f530130194d82912ce2380585680f" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41289 } } peers { permanent_uuid: "10da590321554ba8bc43fe38004fa46c" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45803 } } peers { permanent_uuid: "e3ac260b1f564180a7c66957a311e0ec" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41979 } }
I20260926 02:15:38.114413 8667 consensus_queue.cc:260] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6f530130194d82912ce2380585680f" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41289 } } peers { permanent_uuid: "10da590321554ba8bc43fe38004fa46c" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45803 } } peers { permanent_uuid: "e3ac260b1f564180a7c66957a311e0ec" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41979 } }
I20260926 02:15:38.115010 8666 raft_consensus.cc:348] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6f530130194d82912ce2380585680f" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41289 } } peers { permanent_uuid: "10da590321554ba8bc43fe38004fa46c" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45803 } } peers { permanent_uuid: "e3ac260b1f564180a7c66957a311e0ec" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41979 } }
I20260926 02:15:38.115502 8666 raft_consensus.cc:374] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:15:38.115597 8666 raft_consensus.cc:729] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ac260b1f564180a7c66957a311e0ec, State: Initialized, Role: FOLLOWER
I20260926 02:15:38.116283 8666 consensus_queue.cc:260] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6f530130194d82912ce2380585680f" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41289 } } peers { permanent_uuid: "10da590321554ba8bc43fe38004fa46c" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45803 } } peers { permanent_uuid: "e3ac260b1f564180a7c66957a311e0ec" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41979 } }
I20260926 02:15:38.117616 8511 heartbeater.cc:503] Master 127.28.152.190:46371 was elected leader, sending a full tablet report...
I20260926 02:15:38.118388 8666 ts_tablet_manager.cc:1428] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 02:15:38.122445 8665 ts_tablet_manager.cc:1428] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: Time spent starting tablet: real 0.020s user 0.013s sys 0.005s
I20260926 02:15:38.122969 8646 heartbeater.cc:503] Master 127.28.152.190:46371 was elected leader, sending a full tablet report...
I20260926 02:15:38.126154 8667 ts_tablet_manager.cc:1428] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: Time spent starting tablet: real 0.021s user 0.009s sys 0.007s
W20260926 02:15:38.151430 8512 tablet.cc:2369] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:15:38.211472 8367 tablet.cc:2369] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:15:38.215866 8673 raft_consensus.cc:482] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:15:38.216208 8673 raft_consensus.cc:504] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6f530130194d82912ce2380585680f" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41289 } } peers { permanent_uuid: "10da590321554ba8bc43fe38004fa46c" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45803 } } peers { permanent_uuid: "e3ac260b1f564180a7c66957a311e0ec" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41979 } }
I20260926 02:15:38.218048 8673 leader_election.cc:288] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6f530130194d82912ce2380585680f (127.28.152.131:41289), 10da590321554ba8bc43fe38004fa46c (127.28.152.129:45803)
I20260926 02:15:38.227167 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.218675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:47931 (local address 127.28.152.131:41289)
0926 02:15:38.218851 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:38.218857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:38.219529 (+ 672us) server_negotiation.cc:407] Connection header received
0926 02:15:38.219768 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:38.219773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:38.219852 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:38.219948 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:15:38.220716 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.221846 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.224683 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.226229 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.226393 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.226470 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:38.226790 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 02:15:38.226923 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:15:38.227213 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.218664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:41081 (local address 127.28.152.129:45803)
0926 02:15:38.218851 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:38.218857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:38.219524 (+ 667us) server_negotiation.cc:407] Connection header received
0926 02:15:38.219770 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:38.219775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:38.219852 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:38.219957 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:15:38.220716 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.221847 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.224683 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.226389 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.226526 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.226579 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:38.226855 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 02:15:38.226972 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:15:38.227370 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.218439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:41289 (local address 127.28.152.130:47931)
0926 02:15:38.219439 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0926 02:15:38.219452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 02:15:38.219600 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:15:38.219970 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:15:38.219974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:15:38.220021 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:15:38.220544 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:38.220562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.221982 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.221987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:38.224510 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:38.224527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.226431 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.226436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:38.226626 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.226666 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 02:15:38.226764 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 02:15:38.226904 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":138,"spinlock_wait_cycles":14208,"thread_start_us":646,"threads_started":1}
I20260926 02:15:38.227770 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.218396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45803 (local address 127.28.152.130:41081)
0926 02:15:38.219395 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0926 02:15:38.219415 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:15:38.219628 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:15:38.219970 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:15:38.219974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:15:38.220019 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:15:38.220570 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:38.220579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.222011 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.222016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:38.224510 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:15:38.224527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.226573 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.226580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:15:38.226722 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.226749 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 02:15:38.226821 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 02:15:38.226933 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":199,"spinlock_wait_cycles":17024,"thread_start_us":631,"threads_started":1}
W20260926 02:15:38.227744 8647 tablet.cc:2369] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:15:38.228509 8600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de0c875558d4f92b9fba97e7f5a22f5" candidate_uuid: "e3ac260b1f564180a7c66957a311e0ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6f530130194d82912ce2380585680f" is_pre_election: true
I20260926 02:15:38.228961 8600 raft_consensus.cc:2454] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ac260b1f564180a7c66957a311e0ec in term 0.
I20260926 02:15:38.228894 8321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de0c875558d4f92b9fba97e7f5a22f5" candidate_uuid: "e3ac260b1f564180a7c66957a311e0ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10da590321554ba8bc43fe38004fa46c" is_pre_election: true
I20260926 02:15:38.229349 8321 raft_consensus.cc:2454] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3ac260b1f564180a7c66957a311e0ec in term 0.
I20260926 02:15:38.229627 8401 leader_election.cc:302] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6f530130194d82912ce2380585680f, e3ac260b1f564180a7c66957a311e0ec; no voters:
I20260926 02:15:38.230034 8673 raft_consensus.cc:2781] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:15:38.230162 8673 raft_consensus.cc:482] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:15:38.230249 8673 raft_consensus.cc:3037] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:15:38.232328 8673 raft_consensus.cc:504] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6f530130194d82912ce2380585680f" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41289 } } peers { permanent_uuid: "10da590321554ba8bc43fe38004fa46c" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45803 } } peers { permanent_uuid: "e3ac260b1f564180a7c66957a311e0ec" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41979 } }
I20260926 02:15:38.232882 8673 leader_election.cc:288] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [CANDIDATE]: Term 1 election: Requested vote from peers 5c6f530130194d82912ce2380585680f (127.28.152.131:41289), 10da590321554ba8bc43fe38004fa46c (127.28.152.129:45803)
I20260926 02:15:38.233418 8321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de0c875558d4f92b9fba97e7f5a22f5" candidate_uuid: "e3ac260b1f564180a7c66957a311e0ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10da590321554ba8bc43fe38004fa46c"
I20260926 02:15:38.233409 8600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de0c875558d4f92b9fba97e7f5a22f5" candidate_uuid: "e3ac260b1f564180a7c66957a311e0ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6f530130194d82912ce2380585680f"
I20260926 02:15:38.233611 8600 raft_consensus.cc:3037] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:15:38.233610 8321 raft_consensus.cc:3037] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:15:38.236672 8600 raft_consensus.cc:2454] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3ac260b1f564180a7c66957a311e0ec in term 1.
I20260926 02:15:38.237110 8321 raft_consensus.cc:2454] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3ac260b1f564180a7c66957a311e0ec in term 1.
I20260926 02:15:38.237146 8401 leader_election.cc:302] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6f530130194d82912ce2380585680f, e3ac260b1f564180a7c66957a311e0ec; no voters:
I20260926 02:15:38.237461 8673 raft_consensus.cc:2781] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:15:38.237891 8673 raft_consensus.cc:686] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 1 LEADER]: Becoming Leader. State: Replica: e3ac260b1f564180a7c66957a311e0ec, State: Running, Role: LEADER
I20260926 02:15:38.238364 8673 consensus_queue.cc:237] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6f530130194d82912ce2380585680f" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41289 } } peers { permanent_uuid: "10da590321554ba8bc43fe38004fa46c" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45803 } } peers { permanent_uuid: "e3ac260b1f564180a7c66957a311e0ec" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41979 } }
I20260926 02:15:38.243139 8151 catalog_manager.cc:5495] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec reported cstate change: term changed from 0 to 1, leader changed from <none> to e3ac260b1f564180a7c66957a311e0ec (127.28.152.130). New cstate: current_term: 1 leader_uuid: "e3ac260b1f564180a7c66957a311e0ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6f530130194d82912ce2380585680f" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10da590321554ba8bc43fe38004fa46c" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3ac260b1f564180a7c66957a311e0ec" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 41979 } health_report { overall_health: HEALTHY } } }
I20260926 02:15:38.266264 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.260909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.28.152.130:41979)
0926 02:15:38.261167 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:38.261173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:38.261193 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:15:38.261308 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:38.261312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:38.261399 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:38.261526 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:15:38.261903 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.262619 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.264099 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.264490 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.264597 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.265115 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:15:38.265665 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:15:38.265807 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:38.265911 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 02:15:38.265993 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:15:38.274881 8321 raft_consensus.cc:1261] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c [term 1 FOLLOWER]: Refusing update from remote peer e3ac260b1f564180a7c66957a311e0ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:15:38.274889 8600 raft_consensus.cc:1261] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f [term 1 FOLLOWER]: Refusing update from remote peer e3ac260b1f564180a7c66957a311e0ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:15:38.275878 8673 consensus_queue.cc:1035] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6f530130194d82912ce2380585680f" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:15:38.276250 8676 consensus_queue.cc:1035] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "10da590321554ba8bc43fe38004fa46c" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:15:38.287673 8679 mvcc.cc:205] Tried to move back new op lower bound from 7333433091163967488 to 7333433091028049920. Current Snapshot: MvccSnapshot[applied={T|T < 7333433091163967488}]
I20260926 02:15:38.292464 8680 mvcc.cc:205] Tried to move back new op lower bound from 7333433091163967488 to 7333433091028049920. Current Snapshot: MvccSnapshot[applied={T|T < 7333433091163967488}]
I20260926 02:15:38.293798 8681 mvcc.cc:205] Tried to move back new op lower bound from 7333433091163967488 to 7333433091028049920. Current Snapshot: MvccSnapshot[applied={T|T < 7333433091163967488}]
I20260926 02:15:38.302014 8151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:15:38.302276 8151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:15:38.305925 8151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9546fa39ba644e71b467bc0933a2507e: Sending DeleteTablet for 3 replicas of tablet 0de0c875558d4f92b9fba97e7f5a22f5
I20260926 02:15:38.307090 8580 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de0c875558d4f92b9fba97e7f5a22f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:15:38 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:35418
I20260926 02:15:38.307191 8300 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de0c875558d4f92b9fba97e7f5a22f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:15:38 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:35072
I20260926 02:15:38.307624 8445 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de0c875558d4f92b9fba97e7f5a22f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:15:38 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:32982
I20260926 02:15:38.314452 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 8232
I20260926 02:15:38.318360 8691 tablet_replica.cc:330] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: stopping tablet replica
I20260926 02:15:38.318946 8691 raft_consensus.cc:2229] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:15:38.319761 8691 raft_consensus.cc:2258] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:15:38.320806 8689 tablet_replica.cc:330] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: stopping tablet replica
I20260926 02:15:38.321300 8689 raft_consensus.cc:2229] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:15:38.321812 8689 raft_consensus.cc:2258] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:15:38.323618 8691 ts_tablet_manager.cc:1905] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:15:38.332257 8689 ts_tablet_manager.cc:1905] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:15:38.337360 8691 ts_tablet_manager.cc:1918] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:15:38.337554 8691 log.cc:1196] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/0de0c875558d4f92b9fba97e7f5a22f5
I20260926 02:15:38.338068 8691 ts_tablet_manager.cc:1939] T 0de0c875558d4f92b9fba97e7f5a22f5 P e3ac260b1f564180a7c66957a311e0ec: Deleting consensus metadata
I20260926 02:15:38.339530 8689 ts_tablet_manager.cc:1918] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:15:38.339736 8689 log.cc:1196] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/0de0c875558d4f92b9fba97e7f5a22f5
I20260926 02:15:38.340058 8136 catalog_manager.cc:4841] TS e3ac260b1f564180a7c66957a311e0ec (127.28.152.130:41979): tablet 0de0c875558d4f92b9fba97e7f5a22f5 (table test-table [id=9eebf15f201d4e7c866e2f264c0a7fd5]) successfully deleted
I20260926 02:15:38.340278 8689 ts_tablet_manager.cc:1939] T 0de0c875558d4f92b9fba97e7f5a22f5 P 10da590321554ba8bc43fe38004fa46c: Deleting consensus metadata
I20260926 02:15:38.341637 8690 tablet_replica.cc:330] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: stopping tablet replica
I20260926 02:15:38.342155 8690 raft_consensus.cc:2229] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:15:38.342708 8690 raft_consensus.cc:2258] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:15:38.344942 8690 ts_tablet_manager.cc:1905] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:15:38.346345 8135 catalog_manager.cc:4841] TS 10da590321554ba8bc43fe38004fa46c (127.28.152.129:45803): tablet 0de0c875558d4f92b9fba97e7f5a22f5 (table test-table [id=9eebf15f201d4e7c866e2f264c0a7fd5]) successfully deleted
Sep 26 02:15:38 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:15:38.352576 8690 ts_tablet_manager.cc:1918] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:15:38.352757 8690 log.cc:1196] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/0de0c875558d4f92b9fba97e7f5a22f5
I20260926 02:15:38.353324 8690 ts_tablet_manager.cc:1939] T 0de0c875558d4f92b9fba97e7f5a22f5 P 5c6f530130194d82912ce2380585680f: Deleting consensus metadata
I20260926 02:15:38.353698 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.316289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.28.152.129:45803)
0926 02:15:38.316503 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:38.316510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:38.335504 (+ 18994us) server_negotiation.cc:407] Connection header received
0926 02:15:38.336157 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:38.336164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:38.336277 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:38.336356 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:15:38.338815 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.340107 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.342291 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.342754 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.342877 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.347519 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:15:38.347533 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:15:38.347549 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:15:38.347588 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:15:38.350459 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:38.351280 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:38.351293 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:38.351297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:38.351445 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:38.351834 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:38.351838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:38.351840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:38.352240 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:15:38.352532 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:38.353014 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 02:15:38.353331 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:15:38.354398 8361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:15:38.355333 8138 catalog_manager.cc:4841] TS 5c6f530130194d82912ce2380585680f (127.28.152.131:41289): tablet 0de0c875558d4f92b9fba97e7f5a22f5 (table test-table [id=9eebf15f201d4e7c866e2f264c0a7fd5]) successfully deleted
I20260926 02:15:38.460960 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 8232
I20260926 02:15:38.476096 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 8372
Sep 26 02:15:38 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:15:38.493640 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.477233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.28.152.130:41979)
0926 02:15:38.477411 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:38.477415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:38.477432 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:15:38.477485 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:38.477489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:38.477547 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:38.477640 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:15:38.478985 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.479821 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.481349 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.481771 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.481892 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.486641 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:15:38.486662 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:15:38.486681 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:15:38.486735 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:15:38.490063 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:38.491150 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:38.491159 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:38.491161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:38.491353 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:38.491858 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:38.491868 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:38.491870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:38.492261 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:15:38.492533 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:38.492926 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 02:15:38.493114 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:15:38.494457 8506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:15:38.608937 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 8372
I20260926 02:15:38.624112 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 8517
Sep 26 02:15:38 dist-test-slave-hllj krb5kdc[7671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:15:38.640729 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:15:38.625023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.28.152.131:41289)
0926 02:15:38.625220 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:15:38.625225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:15:38.625247 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:15:38.625370 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:15:38.625375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:15:38.625480 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:15:38.625599 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:15:38.627099 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.627823 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.629318 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:15:38.629733 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:15:38.629841 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:15:38.634829 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:15:38.634848 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:15:38.634867 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:15:38.634916 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:15:38.637951 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:38.638813 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:38.638821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:38.638823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:38.638970 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:15:38.639367 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:15:38.639370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:15:38.639372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:15:38.639744 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:15:38.639950 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 02:15:38.640298 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 02:15:38.640469 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:15:38.641392 8641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:15:38.731236 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 8517
I20260926 02:15:38.744098 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 8119
I20260926 02:15:38.745378 8182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:15:38.857256 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 8119
I20260926 02:15:38.876713 29282 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 02:15:39.212773 29282 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 02:15:39.212984 29282 mini_ranger.cc:66] Stopping Ranger...
I20260926 02:15:39.549008 29282 mini_ranger.cc:68] Stopped Ranger
2026-09-26 02:15:39.549 UTC [7701] LOG: received smart shutdown request
2026-09-26 02:15:39.553 UTC [7701] LOG: background worker "logical replication launcher" (PID 7710) exited with exit code 1
2026-09-26 02:15:39.553 UTC [7705] LOG: shutting down
2026-09-26 02:15:39.726 UTC [7701] LOG: database system is shut down
2026-09-26T02:15:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50396 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:15:39 dist-test-slave-hllj krb5kdc[8705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:15:39 dist-test-slave-hllj krb5kdc[8705](info): set up 2 sockets
Sep 26 02:15:39 dist-test-slave-hllj krb5kdc[8705](info): commencing operation
krb5kdc: starting...
W20260926 02:15:40.866704 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:15:41 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:15:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:15:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.152.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.152.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/rangeradmin_127.28.152.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/rangeradmin_127.28.152.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.152.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.152.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/rangerlookup_127.28.152.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/rangerlookup_127.28.152.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.152.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.152.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/HTTP_127.28.152.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/HTTP_127.28.152.148@KRBTEST.COM.keytab.
I20260926 02:15:41.188330 29282 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCd4s0O/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/postgres -l logfile start
2026-09-26 02:15:43.784 UTC [8735] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 02:15:43.784 UTC [8735] LOG: listening on IPv4 address "127.28.152.148", port 47535
2026-09-26 02:15:43.787 UTC [8735] LOG: listening on Unix socket "/tmp/.s.PGSQL.47535"
2026-09-26 02:15:43.809 UTC [8738] LOG: database system was shut down at 2026-09-26 02:15:42 UTC
2026-09-26 02:15:43.813 UTC [8735] LOG: database system is ready to accept connections
I20260926 02:15:44.694262 29282 mini_postgres.cc:96] Postgres bound to 47535
2026-09-26 02:15:44.714 UTC [8746] FATAL: database "slave" does not exist
127.28.152.148:47535 - accepting connections
I20260926 02:15:44.715018 29282 mini_ranger.cc:161] Starting Ranger...
I20260926 02:15:44.740906 29282 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 02:15:45.259183 29282 mini_ranger.cc:87] Created ranger Postgres database
I20260926 02:15:45.259287 29282 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-admin
2026-09-26 02:15:45,571 [I] DB FLAVOR :POSTGRES
2026-09-26 02:15:45,571 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:15:45,571 [I] Checking connection..
2026-09-26 02:15:45,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 02:15:45,833 [I] Checking connection passed.
2026-09-26 02:15:45,834 [I] --------- Verifying version history table ---------
2026-09-26 02:15:45,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:15:46,071 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:15:46,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:15:46,335 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:15:46,335 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 02:15:46,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 02:15:46.586 UTC [8811] WARNING: there is no transaction in progress
2026-09-26 02:15:46,592 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 02:15:46,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:15:46,856 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 02:15:46,856 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:15:46,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 02:15:47,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 02:15:47,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hllj.c.gcp-upstream.internal','N') ;"
2026-09-26 02:15:47,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:15:47,860 [I] Table x_portal_user does not exist in database ranger
2026-09-26 02:15:47,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:15:48,109 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 02:15:48,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:15:48,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:15:48,605 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 02:15:48,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 02:15:49.607 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.623 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.640 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.656 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.677 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.698 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.766 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.774 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.791 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.804 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.817 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.832 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.844 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.851 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.859 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:49.866 UTC [8920] WARNING: there is no transaction in progress
2026-09-26 02:15:50,589 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 02:15:50,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 02:15:50,849 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 02:15:50,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:15:51,113 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 02:15:51,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:15:51,362 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 02:15:51,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:15:51,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:15:51,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hllj.c.gcp-upstream.internal';"
2026-09-26 02:15:52,130 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 02:15:52,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 02:15:52,364 [I] DB_PATCHES have already been applied
I20260926 02:15:52.370590 29282 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCd4s0O/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 02:15:52.370682 29282 mini_ranger.cc:193] Using host: 127.28.152.148
I20260926 02:15:52.384460 29282 mini_ranger.cc:227] Ranger admin URL: http://127.28.152.148:41419
Sep 26, 2026 2:15:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:15:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:15:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:15:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:15:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:15:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:15:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 02:15:53.427853 29282 mini_ranger.cc:160] Time spent starting Ranger: real 8.713s user 0.004s sys 0.036s
Sep 26, 2026 2:16:14 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:16:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 2:16:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:16:14 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:16:15 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 02:16:15.893373 29282 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.152.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.152.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/rangerkms_127.28.152.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.152.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/rangerkms_127.28.152.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.152.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.152.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.152.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/HTTP_127.28.152.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.152.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/HTTP_127.28.152.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 02:16:16 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388976, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 02:16:16.064683 29282 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 02:16:16.087352 29282 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 02:16:16.994971 29282 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 02:16:16.995085 29282 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-kms
I20260926 02:16:16.995133 29282 mini_ranger_kms.cc:227] Using postgres at 127.28.152.148:47535
2026-09-26 02:16:17,346 [I] DB FLAVOR :POSTGRES
2026-09-26 02:16:17,346 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:16:17,346 [I] Checking connection
2026-09-26 02:16:17,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 02:16:17,597 [I] connection success
2026-09-26 02:16:17,597 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 02:16:17,597 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 02:16:17,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 02:16:17,877 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 02:16:17,877 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:16:17,878 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 02:16:17,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.152.148:47535/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 02:16:18,205 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 02:16:18.672374 29282 mini_ranger_kms.cc:310] Created kms service
I20260926 02:16:18.886883 29282 mini_ranger_kms.cc:326] Created kudu user
I20260926 02:16:18.933701 29282 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 02:16:18 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388976, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:16:19.346364 29282 mini_ranger_kms.cc:384] Added ranger policy
I20260926 02:16:19.346689 29282 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCd4s0O/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCd4s0O/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 02:16:19.346747 29282 mini_ranger_kms.cc:240] Using host: 127.28.152.148
I20260926 02:16:19.361619 29282 mini_ranger_kms.cc:276] Ranger KMS PID: 9103
I20260926 02:16:19.361763 29282 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.152.148:50921
Sep 26, 2026 2:16:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:16:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:16:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:16:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:16:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:16:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 02:16:21.489717 29282 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.425s user 0.067s sys 0.297s
I20260926 02:16:21.489970 29282 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 02:16:21.490125 29282 mini_ranger_kms.cc:401] 127.28.152.148:50921/kms/v1/keys
Sep 26 02:16:21 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790388981, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.152.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 02:16:22 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388981, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.152.148@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
Sep 26 02:16:22 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388981, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.152.148@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
Sep 26 02:16:25 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:16:25.097170 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:42877
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:42877
--ranger_config_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:50921/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:25.436743 9153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:25.437086 9153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:25.437248 9153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:25.444703 9153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:16:25.444846 9153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:25.444921 9153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:25.445005 9153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:16:25.445065 9153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:16:25.455497 9153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.152.148:50921/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:42877
--ranger_config_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:25.456876 9153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:25.458326 9153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:25.471045 9159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:25.471055 9158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:25.482791 9161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:25.484774 9153 server_base.cc:1034] running on GCE node
I20260926 02:16:25.486214 9153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:25.488410 9153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:25.489734 9153 hybrid_clock.cc:648] HybridClock initialized: now 1790388985489670 us; error 53 us; skew 500 ppm
I20260926 02:16:25.490227 9153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:25 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388985, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:25.501812 9153 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:16:25.504043 9153 webserver.cc:460] Webserver started at http://127.28.152.190:37809/ using document root <none> and password file <none>
I20260926 02:16:25.505015 9153 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:25.505170 9153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:25.505599 9153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:16:25 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388985, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:16:25.574658 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "7e31d55439f24b3085a148e25cdcd779"
format_stamp: "Formatted at 2026-09-26 02:16:25 on dist-test-slave-hllj"
server_key: "d6bd448b529c030e1bacac75956a7ff3"
server_key_iv: "322f667b0f12df41b7dbe9f0a88d56f6"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:25.576027 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "7e31d55439f24b3085a148e25cdcd779"
format_stamp: "Formatted at 2026-09-26 02:16:25 on dist-test-slave-hllj"
server_key: "d6bd448b529c030e1bacac75956a7ff3"
server_key_iv: "322f667b0f12df41b7dbe9f0a88d56f6"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:25.584101 9153 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260926 02:16:25.604365 9168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:25.606002 9153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:16:25.606194 9153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "7e31d55439f24b3085a148e25cdcd779"
format_stamp: "Formatted at 2026-09-26 02:16:25 on dist-test-slave-hllj"
server_key: "d6bd448b529c030e1bacac75956a7ff3"
server_key_iv: "322f667b0f12df41b7dbe9f0a88d56f6"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:25.606413 9153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:25.621515 9153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:25.625540 9153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:25.625833 9153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:25.644481 9153 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:42877
I20260926 02:16:25.644483 9219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:42877 every 8 connection(s)
I20260926 02:16:25.645931 9153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:16:25.649897 9220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:25.656176 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 9153
I20260926 02:16:25.656414 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:16:25.657415 29282 external_mini_cluster.cc:1428] Setting key fc976ea178b629243186865fbf4055d9
I20260926 02:16:25.661507 9220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779: Bootstrap starting.
I20260926 02:16:25.667555 9220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:25.669358 9220 log.cc:824] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:25.678171 9220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779: No bootstrap required, opened a new log
Sep 26 02:16:25 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:25.689443 9220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } }
I20260926 02:16:25.689972 9220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:25.690135 9220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e31d55439f24b3085a148e25cdcd779, State: Initialized, Role: FOLLOWER
I20260926 02:16:25.690881 9220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } }
I20260926 02:16:25.691148 9220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:16:25.691323 9220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:16:25.691529 9220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:25.694954 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:25.659807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.28.152.190:42877)
0926 02:16:25.675247 (+ 15440us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:25.675266 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:25.675327 (+ 61us) server_negotiation.cc:407] Connection header received
0926 02:16:25.677210 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:25.677238 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:25.677678 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:25.678085 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:25.679550 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:25.680989 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:25.682438 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:25.683183 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:25.683426 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:25.687437 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:25.687450 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:25.687470 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:25.687526 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:25.690616 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:25.691400 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:25.691409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:25.691418 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:25.691615 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:25.691997 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:25.692002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:25.692005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:25.692584 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:25.692879 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:25.693409 (+ 530us) server_negotiation.cc:299] Negotiation successful
0926 02:16:25.693889 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15149,"thread_start_us":14923,"threads_started":1}
I20260926 02:16:25.694989 9220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } }
I20260926 02:16:25.695755 9220 leader_election.cc:302] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e31d55439f24b3085a148e25cdcd779; no voters:
I20260926 02:16:25.696517 9220 leader_election.cc:288] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:16:25.696599 9225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:16:25.699368 9225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 1 LEADER]: Becoming Leader. State: Replica: 7e31d55439f24b3085a148e25cdcd779, State: Running, Role: LEADER
I20260926 02:16:25.699988 9225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } }
I20260926 02:16:25.701784 9225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e31d55439f24b3085a148e25cdcd779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } } }
I20260926 02:16:25.702096 9225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:25.703622 9227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e31d55439f24b3085a148e25cdcd779. Latest consensus state: current_term: 1 leader_uuid: "7e31d55439f24b3085a148e25cdcd779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } } }
I20260926 02:16:25.703992 9227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:25.705994 9220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:16:25.717973 9229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:16:25.718288 9220 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 02:16:25.723453 9229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:16:25.757894 9229 catalog_manager.cc:1347] Generated new cluster ID: 80c6cba9ebda41248813d70804139d3a
I20260926 02:16:25.758061 9229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:16:25.814253 9229 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:16:25.816414 9229 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:16:25.831559 9229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779: Generated new TSK 0
I20260926 02:16:25.832826 9229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:16:26.662966 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:50921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42877
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:16:26 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790388986, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 02:16:27.053740 9266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:27.054175 9266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:27.054332 9266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:27.062610 9266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:27.062752 9266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:27.062953 9266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:16:27.073644 9266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.152.148:50921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:27.075276 9266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:27.076993 9266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:27.095618 9277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:27.095758 9279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:27.096583 9276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:27.096949 9266 server_base.cc:1034] running on GCE node
I20260926 02:16:27.097995 9266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:27.099472 9266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:27.100809 9266 hybrid_clock.cc:648] HybridClock initialized: now 1790388987100742 us; error 65 us; skew 500 ppm
I20260926 02:16:27.101253 9266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:27 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388987, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:27.105535 9266 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:16:27.113826 9266 webserver.cc:460] Webserver started at http://127.28.152.129:34973/ using document root <none> and password file <none>
I20260926 02:16:27.114928 9266 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:27.115149 9266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:27.115593 9266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:16:27 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388987, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:16:27.142064 9266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "5a6709a5f1f14a968e988a8e5536bd41"
format_stamp: "Formatted at 2026-09-26 02:16:27 on dist-test-slave-hllj"
server_key: "e142cacd34f67d835ea4ae3164899285"
server_key_iv: "ed51fd16ac843cb0af08b3132a584df6"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:27.143445 9266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "5a6709a5f1f14a968e988a8e5536bd41"
format_stamp: "Formatted at 2026-09-26 02:16:27 on dist-test-slave-hllj"
server_key: "e142cacd34f67d835ea4ae3164899285"
server_key_iv: "ed51fd16ac843cb0af08b3132a584df6"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:27.156615 9266 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.003s sys 0.008s
I20260926 02:16:27.189875 9286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:27.193022 9266 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.012s sys 0.000s
I20260926 02:16:27.193291 9266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "5a6709a5f1f14a968e988a8e5536bd41"
format_stamp: "Formatted at 2026-09-26 02:16:27 on dist-test-slave-hllj"
server_key: "e142cacd34f67d835ea4ae3164899285"
server_key_iv: "ed51fd16ac843cb0af08b3132a584df6"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:27.193599 9266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:27.212292 9266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:27.217738 9266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:27.218057 9266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:27.219234 9266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:27.220975 9266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:27.221078 9266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:27.221220 9266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:27.221280 9266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:27.280450 9266 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:46135
I20260926 02:16:27.280409 9399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:46135 every 8 connection(s)
I20260926 02:16:27.282411 9266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:16:27.291302 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 9266
I20260926 02:16:27.291530 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:16:27.292312 29282 external_mini_cluster.cc:1428] Setting key cb68e0e71edc57a9748e841b4ea3b8af
Sep 26 02:16:27 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388987, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
I20260926 02:16:27.329937 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:27.292434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:42877 (local address 127.28.152.129:46799)
0926 02:16:27.303220 (+ 10786us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:27.303294 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:27.305825 (+ 2531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:27.308516 (+ 2691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:27.308532 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:27.309304 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:27.310308 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:27.310347 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:27.312110 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:27.312116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:27.314396 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:27.314418 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:27.315891 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:27.315901 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:27.316261 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:27.317227 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:27.317260 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:27.320163 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:27.324558 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:27.324570 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:27.324592 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:27.325341 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:27.325871 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:27.325876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:27.325880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:27.326175 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:27.326956 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:27.326966 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:27.327475 (+ 509us) client_negotiation.cc:769] Sending connection context
0926 02:16:27.327794 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 02:16:27.328396 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10425,"thread_start_us":13227,"threads_started":1}
I20260926 02:16:27.331231 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:27.291779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:46799 (local address 127.28.152.190:42877)
0926 02:16:27.307513 (+ 15734us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:27.307520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:27.307546 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:16:27.307616 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:27.307622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:27.307727 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:27.307864 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:27.310627 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:27.311884 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:27.314724 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:27.315525 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:27.316769 (+ 1244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:27.320609 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:27.320617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:27.320622 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:27.320672 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:27.324306 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:27.325533 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:27.325540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:27.325544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:27.325728 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:27.326379 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:27.326386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:27.326390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:27.326793 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:27.326959 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:27.330559 (+ 3600us) server_negotiation.cc:299] Negotiation successful
0926 02:16:27.330828 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15573,"thread_start_us":16183,"threads_started":1}
I20260926 02:16:27.334182 9400 heartbeater.cc:347] Connected to a master server at 127.28.152.190:42877
I20260926 02:16:27.334543 9400 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:27.335440 9400 heartbeater.cc:511] Master 127.28.152.190:42877 requested a full tablet report, sending...
I20260926 02:16:27.337975 9185 ts_manager.cc:194] Registered new tserver with Master: 5a6709a5f1f14a968e988a8e5536bd41 (127.28.152.129:46135)
I20260926 02:16:27.340013 9185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:46799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:16:27.361294 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:50921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42877
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:16:27 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388986, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
Sep 26 02:16:27 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388986, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
W20260926 02:16:27.829227 9406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:27.829726 9406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:27.829908 9406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:27.840867 9406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:27.841169 9406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:27.841441 9406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:16:27.856276 9406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.152.148:50921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:27.857803 9406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:27.859237 9406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:27.879163 9413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:27.879719 9414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:27.905680 9406 server_base.cc:1034] running on GCE node
W20260926 02:16:27.907060 9416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:27.909309 9406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:27.925475 9406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:27.926869 9406 hybrid_clock.cc:648] HybridClock initialized: now 1790388987926810 us; error 71 us; skew 500 ppm
I20260926 02:16:27.927402 9406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:27 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388987, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:27.932168 9406 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:16:27.950330 9406 webserver.cc:460] Webserver started at http://127.28.152.130:45205/ using document root <none> and password file <none>
I20260926 02:16:27.951611 9406 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:27.951884 9406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:27.952363 9406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:16:27 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388987, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:16:27.991997 9406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "ffd364faeeb24c99917db63dc664902c"
format_stamp: "Formatted at 2026-09-26 02:16:27 on dist-test-slave-hllj"
server_key: "acc5f36dc3bbe80803ea8816ed1af122"
server_key_iv: "41a9f2df0e317b242b889807144eb48e"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:27.993213 9406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "ffd364faeeb24c99917db63dc664902c"
format_stamp: "Formatted at 2026-09-26 02:16:27 on dist-test-slave-hllj"
server_key: "acc5f36dc3bbe80803ea8816ed1af122"
server_key_iv: "41a9f2df0e317b242b889807144eb48e"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:28.024941 9406 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.003s sys 0.006s
I20260926 02:16:28.069836 9424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:28.078511 9406 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.004s sys 0.000s
I20260926 02:16:28.078789 9406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "ffd364faeeb24c99917db63dc664902c"
format_stamp: "Formatted at 2026-09-26 02:16:27 on dist-test-slave-hllj"
server_key: "acc5f36dc3bbe80803ea8816ed1af122"
server_key_iv: "41a9f2df0e317b242b889807144eb48e"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:28.079082 9406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:28.141678 9406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:28.149477 9406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:28.149840 9406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:28.161661 9406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:28.174124 9406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:28.174346 9406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:16:28.174549 9406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:28.174695 9406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:28.215080 9406 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:38489
I20260926 02:16:28.215103 9543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:38489 every 8 connection(s)
I20260926 02:16:28.216665 9406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:16:28.224148 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 9406
I20260926 02:16:28.224335 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:16:28.224852 29282 external_mini_cluster.cc:1428] Setting key 86efd947e991c22229c0a23cc730db08
Sep 26 02:16:28 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388987, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:16:28.260640 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:28.221214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:32927 (local address 127.28.152.190:42877)
0926 02:16:28.229547 (+ 8333us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:28.229554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:28.229577 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:16:28.229640 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:28.229646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:28.229738 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:28.229865 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:28.232507 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.233697 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.241626 (+ 7929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.242254 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.242365 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:28.253245 (+ 10880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:28.253253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:28.253258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:28.253313 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:28.256960 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:28.258042 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:28.258047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:28.258049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:28.258181 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:28.258720 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:28.258724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:28.258727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:28.259024 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:28.259152 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:28.259972 (+ 820us) server_negotiation.cc:299] Negotiation successful
0926 02:16:28.260216 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8192,"thread_start_us":8073,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
I20260926 02:16:28.261898 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:28.220585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:42877 (local address 127.28.152.130:32927)
0926 02:16:28.221572 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:28.221635 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:28.224199 (+ 2564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:28.230406 (+ 6207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:28.230423 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:28.231165 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:28.232235 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:28.232267 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.237930 (+ 5663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.237938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:28.240458 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:28.240483 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.242732 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.242737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:28.242954 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:28.250465 (+ 7511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:28.250494 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:28.252903 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:28.257287 (+ 4384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:28.257300 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:28.257324 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:28.257868 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:28.258321 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:28.258327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:28.258330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:28.258586 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:28.259185 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:28.259195 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:28.259559 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 02:16:28.259841 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 02:16:28.260416 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":376,"threads_started":1}
I20260926 02:16:28.263792 9544 heartbeater.cc:347] Connected to a master server at 127.28.152.190:42877
I20260926 02:16:28.264150 9544 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:28.265034 9544 heartbeater.cc:511] Master 127.28.152.190:42877 requested a full tablet report, sending...
I20260926 02:16:28.266863 9185 ts_manager.cc:194] Registered new tserver with Master: ffd364faeeb24c99917db63dc664902c (127.28.152.130:38489)
I20260926 02:16:28.267894 9185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:32927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:16:28.296988 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:50921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42877
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:16:28.344125 9400 heartbeater.cc:503] Master 127.28.152.190:42877 was elected leader, sending a full tablet report...
W20260926 02:16:28.640237 9550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:28.640559 9550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:28.640682 9550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:28.650966 9550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:28.651125 9550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:28.651350 9550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:16:28.661423 9550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.152.148:50921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:28.662767 9550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:28.664109 9550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:28.677052 9558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:28.677145 9556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:28.677105 9555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:28.677726 9550 server_base.cc:1034] running on GCE node
I20260926 02:16:28.678340 9550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:28.679349 9550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:28.680606 9550 hybrid_clock.cc:648] HybridClock initialized: now 1790388988680534 us; error 60 us; skew 500 ppm
I20260926 02:16:28.680977 9550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:28 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388988, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:28.684540 9550 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:16:28.686246 9550 webserver.cc:460] Webserver started at http://127.28.152.131:40921/ using document root <none> and password file <none>
I20260926 02:16:28.687018 9550 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:28.687146 9550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:28.687467 9550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:16:28 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388988, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:16:28.703987 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "0d538681c5224242a96ef5b442d17edd"
format_stamp: "Formatted at 2026-09-26 02:16:28 on dist-test-slave-hllj"
server_key: "b43980a9918c0c61ac23717207e99e1c"
server_key_iv: "b7bb6010c34fa2084e59205db5a36ec0"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:28.704821 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "0d538681c5224242a96ef5b442d17edd"
format_stamp: "Formatted at 2026-09-26 02:16:28 on dist-test-slave-hllj"
server_key: "b43980a9918c0c61ac23717207e99e1c"
server_key_iv: "b7bb6010c34fa2084e59205db5a36ec0"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:28.710525 9550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:16:28.728298 9565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:28.730190 9550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:16:28.730382 9550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "0d538681c5224242a96ef5b442d17edd"
format_stamp: "Formatted at 2026-09-26 02:16:28 on dist-test-slave-hllj"
server_key: "b43980a9918c0c61ac23717207e99e1c"
server_key_iv: "b7bb6010c34fa2084e59205db5a36ec0"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:28.730635 9550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:28.747781 9550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:28.751892 9550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:28.752163 9550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:28.753101 9550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:28.754597 9550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:28.754686 9550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:28.754820 9550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:28.754879 9550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:28.785337 9550 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:33583
I20260926 02:16:28.785369 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:33583 every 8 connection(s)
I20260926 02:16:28.786718 9550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:16:28.795950 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 9550
I20260926 02:16:28.796128 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:16:28.796756 29282 external_mini_cluster.cc:1428] Setting key 9e13aa83bba6264b86095b582dc3b436
Sep 26 02:16:28 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388988, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:28.806720 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:28.789098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:59107 (local address 127.28.152.190:42877)
0926 02:16:28.789569 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:28.789573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:28.790489 (+ 916us) server_negotiation.cc:407] Connection header received
0926 02:16:28.792269 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:28.792276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:28.792354 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:28.792450 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:28.793983 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.794714 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.796773 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.797296 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.797429 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:28.800937 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:28.800943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:28.800946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:28.800982 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:28.803477 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:28.804249 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:28.804254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:28.804256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:28.804391 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:28.804875 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:28.804879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:28.804882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:28.805220 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:28.805373 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:28.806100 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 02:16:28.806341 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":196,"threads_started":1}
I20260926 02:16:28.807739 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:28.789617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:42877 (local address 127.28.152.131:59107)
0926 02:16:28.790320 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:28.790366 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:28.791965 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:28.792709 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:28.792718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:28.793192 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:28.793801 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:28.793819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.794872 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.794876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:28.796585 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:28.796601 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.797550 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.797559 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:28.797809 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:28.798505 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:28.798529 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:28.800712 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:28.803645 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:28.803665 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:28.803681 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:28.804122 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:28.804507 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:28.804512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:28.804516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:28.804759 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:28.805346 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:28.805354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:28.805724 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 02:16:28.805974 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 02:16:28.806508 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":275,"threads_started":1}
I20260926 02:16:28.809188 9679 heartbeater.cc:347] Connected to a master server at 127.28.152.190:42877
I20260926 02:16:28.809547 9679 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:28.810356 9679 heartbeater.cc:511] Master 127.28.152.190:42877 requested a full tablet report, sending...
I20260926 02:16:28.812085 9185 ts_manager.cc:194] Registered new tserver with Master: 0d538681c5224242a96ef5b442d17edd (127.28.152.131:33583)
I20260926 02:16:28.812882 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:16:28.813184 9185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:59107
I20260926 02:16:28.815029 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 9266
Sep 26 02:16:28 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:16:28.831526 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:28.816007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.28.152.129:46135)
0926 02:16:28.816612 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:28.816623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:28.816649 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:16:28.816742 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:28.816749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:28.816950 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:28.817082 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:28.818586 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.819592 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.821007 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.821699 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.821801 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:28.825943 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:28.825957 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:28.825972 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:28.826021 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:28.828888 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:28.829669 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:28.829677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:28.829679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:28.829827 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:28.830182 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:28.830185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:28.830187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:28.830510 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:28.830732 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:28.830971 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 02:16:28.831177 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":327,"threads_started":1}
I20260926 02:16:28.832811 9395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:28.915688 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 9266
I20260926 02:16:28.928596 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 9406
Sep 26 02:16:28 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:16:28.946353 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:28.929643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.28.152.130:38489)
0926 02:16:28.930131 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:28.930139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:28.930165 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:16:28.930226 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:28.930232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:28.930439 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:28.930599 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:28.931830 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.932813 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.934479 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:28.935086 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:28.935198 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:28.939998 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:28.940016 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:28.940034 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:28.940082 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:28.942990 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:28.943862 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:28.943871 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:28.943874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:28.944112 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:28.944577 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:28.944581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:28.944584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:28.944969 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:28.945327 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:28.945703 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 02:16:28.945948 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":236,"threads_started":1}
I20260926 02:16:28.947630 9539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:29.028679 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 9406
I20260926 02:16:29.043465 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 9550
Sep 26 02:16:29 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:16:29.060003 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:29.044454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.28.152.131:33583)
0926 02:16:29.045096 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:29.045103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:29.045150 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:16:29.045225 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:29.045232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:29.045438 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:29.045574 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:29.046893 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:29.047931 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:29.049386 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:29.049974 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:29.050083 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:29.054527 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:29.054543 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:29.054560 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:29.054603 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:29.057457 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:29.058195 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:29.058203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:29.058206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:29.058345 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:29.058651 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:29.058654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:29.058656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:29.058989 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:29.059218 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:29.059497 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 02:16:29.059712 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":355,"threads_started":1}
I20260926 02:16:29.061604 9674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:29.143567 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 9550
I20260926 02:16:29.155798 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 9153
I20260926 02:16:29.156996 9215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:29.260890 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 9153
I20260926 02:16:29.282245 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:42877
--webserver_interface=127.28.152.190
--webserver_port=37809
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:42877
--ranger_config_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:50921/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.152.148:50921/kms with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:29.611769 9691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:29.612084 9691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:29.612217 9691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:29.619704 9691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:16:29.619848 9691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:29.619915 9691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:29.619987 9691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:16:29.620054 9691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:16:29.630314 9691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.152.148:50921/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:42877
--ranger_config_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=37809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:29.631664 9691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:29.632979 9691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:29.644834 9697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:29.644860 9696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:29.644960 9699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:29.645581 9691 server_base.cc:1034] running on GCE node
I20260926 02:16:29.646422 9691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:29.647926 9691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:29.649255 9691 hybrid_clock.cc:648] HybridClock initialized: now 1790388989649210 us; error 37 us; skew 500 ppm
I20260926 02:16:29.649642 9691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:29 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388989, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:29.652913 9691 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:16:29.654556 9691 webserver.cc:460] Webserver started at http://127.28.152.190:37809/ using document root <none> and password file <none>
I20260926 02:16:29.655252 9691 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:29.655378 9691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:16:29 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388989, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:16:29.703917 9691 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:16:29.707731 9707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:29.709262 9691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:16:29.709444 9691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "7e31d55439f24b3085a148e25cdcd779"
format_stamp: "Formatted at 2026-09-26 02:16:25 on dist-test-slave-hllj"
server_key: "d6bd448b529c030e1bacac75956a7ff3"
server_key_iv: "322f667b0f12df41b7dbe9f0a88d56f6"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:29.710419 9691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:29.724174 9691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:29.728327 9691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:29.728623 9691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:29.746567 9691 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:42877
I20260926 02:16:29.746587 9758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:42877 every 8 connection(s)
I20260926 02:16:29.748152 9691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:16:29.755237 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 9691
I20260926 02:16:29.755925 9759 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:16:29.756347 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:50921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:46135
--local_ip_for_outbound_sockets=127.28.152.129
--tserver_master_addrs=127.28.152.190:42877
--webserver_port=34973
--webserver_interface=127.28.152.129
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.152.148:50921/kms with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:16:29.760881 9759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779: Bootstrap starting.
I20260926 02:16:29.782109 9759 log.cc:824] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:29.792438 9759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:16:29.793028 9759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779: Bootstrap complete.
I20260926 02:16:29.802609 9759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } }
I20260926 02:16:29.803525 9759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e31d55439f24b3085a148e25cdcd779, State: Initialized, Role: FOLLOWER
I20260926 02:16:29.804198 9759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } }
I20260926 02:16:29.804483 9759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:16:29.804606 9759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:16:29.804812 9759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:16:29.809011 9759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } }
I20260926 02:16:29.809664 9759 leader_election.cc:302] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e31d55439f24b3085a148e25cdcd779; no voters:
I20260926 02:16:29.810303 9759 leader_election.cc:288] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:16:29.810467 9763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:16:29.811533 9763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [term 2 LEADER]: Becoming Leader. State: Replica: 7e31d55439f24b3085a148e25cdcd779, State: Running, Role: LEADER
I20260926 02:16:29.812256 9763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } }
I20260926 02:16:29.812762 9759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:16:29.815191 9765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e31d55439f24b3085a148e25cdcd779. Latest consensus state: current_term: 2 leader_uuid: "7e31d55439f24b3085a148e25cdcd779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } } }
I20260926 02:16:29.815589 9765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:29.816192 9764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7e31d55439f24b3085a148e25cdcd779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e31d55439f24b3085a148e25cdcd779" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42877 } } }
I20260926 02:16:29.816476 9764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:29.825773 9769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:16:29.832581 9769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:16:29.834443 9769 catalog_manager.cc:1259] Loaded cluster ID: 80c6cba9ebda41248813d70804139d3a
I20260926 02:16:29.834702 9769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:16:29.840529 9769 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:16:29.842299 9759 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 02:16:29.842409 9769 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779: Loaded TSK: 0
I20260926 02:16:29.844386 9769 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:16:30.241421 9761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:30.241935 9761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:30.242136 9761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:30.253880 9761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:30.254027 9761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:30.254257 9761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:16:30.269912 9761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.152.148:50921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:46135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=34973
--tserver_master_addrs=127.28.152.190:42877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:30.271746 9761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:30.273691 9761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:30.336378 9797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:30.337364 9799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:30.339697 9796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:30.341430 9761 server_base.cc:1034] running on GCE node
I20260926 02:16:30.342370 9761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:30.353468 9761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:30.357087 9761 hybrid_clock.cc:648] HybridClock initialized: now 1790388990356980 us; error 95 us; skew 500 ppm
I20260926 02:16:30.357822 9761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:30 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388990, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:30.361335 9761 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:16:30.391582 9761 webserver.cc:460] Webserver started at http://127.28.152.129:34973/ using document root <none> and password file <none>
I20260926 02:16:30.392745 9761 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:30.392969 9761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:16:30 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388990, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:16:30.419395 9761 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 02:16:30.427860 9808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:30.429643 9761 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.004s
I20260926 02:16:30.429888 9761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "5a6709a5f1f14a968e988a8e5536bd41"
format_stamp: "Formatted at 2026-09-26 02:16:27 on dist-test-slave-hllj"
server_key: "e142cacd34f67d835ea4ae3164899285"
server_key_iv: "ed51fd16ac843cb0af08b3132a584df6"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:30.431031 9761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:30.471474 9761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:30.476989 9761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:30.477437 9761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:30.481557 9761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:30.486676 9761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:30.486825 9761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:30.486963 9761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:30.487053 9761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:30.546980 9761 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:46135
I20260926 02:16:30.547053 9920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:46135 every 8 connection(s)
I20260926 02:16:30.549170 9761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:16:30.557152 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 9761
I20260926 02:16:30.558045 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:50921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:38489
--local_ip_for_outbound_sockets=127.28.152.130
--tserver_master_addrs=127.28.152.190:42877
--webserver_port=45205
--webserver_interface=127.28.152.130
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.152.148:50921/kms with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:16:30 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388990, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:30.618995 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:30.552710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:42877 (local address 127.28.152.129:40199)
0926 02:16:30.581950 (+ 29240us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:30.582009 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:30.584316 (+ 2307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:30.587511 (+ 3195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:30.587522 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:30.588167 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:30.589042 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:30.589095 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:30.591656 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:30.591662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:30.593918 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:30.593953 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:30.596175 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:30.596185 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:30.596524 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:30.604371 (+ 7847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:30.604420 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:30.607537 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:30.613262 (+ 5725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:30.613273 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:30.613293 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:30.614101 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:30.614843 (+ 742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:30.614849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:30.614853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:30.615136 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:30.616360 (+ 1224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:30.616368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:30.616759 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 02:16:30.617063 (+ 304us) client_negotiation.cc:240] Negotiation successful
0926 02:16:30.617687 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28941,"thread_start_us":28760,"threads_started":1}
I20260926 02:16:30.620603 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:30.552568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:40199 (local address 127.28.152.190:42877)
0926 02:16:30.581285 (+ 28717us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:30.581312 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:30.585103 (+ 3791us) server_negotiation.cc:407] Connection header received
0926 02:16:30.585658 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:30.585680 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:30.586248 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:30.586725 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:30.589403 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:30.591375 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:30.594209 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:30.595463 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:30.595710 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:30.608032 (+ 12322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:30.608048 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:30.608075 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:30.608153 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:30.612984 (+ 4831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:30.614350 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:30.614360 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:30.614369 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:30.614649 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:30.615421 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:30.615427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:30.615430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:30.616081 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:30.616283 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:30.618706 (+ 2423us) server_negotiation.cc:299] Negotiation successful
0926 02:16:30.619246 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28300,"thread_start_us":32040,"threads_started":1}
I20260926 02:16:30.624769 9921 heartbeater.cc:347] Connected to a master server at 127.28.152.190:42877
I20260926 02:16:30.625161 9921 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:30.626145 9921 heartbeater.cc:511] Master 127.28.152.190:42877 requested a full tablet report, sending...
W20260926 02:16:30.630079 9921 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:42877 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:16:30.630357 9921 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:30.630698 9921 heartbeater.cc:511] Master 127.28.152.190:42877 requested a full tablet report, sending...
I20260926 02:16:30.632266 9921 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:30.632603 9921 heartbeater.cc:511] Master 127.28.152.190:42877 requested a full tablet report, sending...
W20260926 02:16:30.633968 9921 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:16:30.980574 9925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:30.980906 9925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:30.981089 9925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:30.990418 9925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:30.990661 9925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:30.990891 9925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:16:31.007283 9925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.152.148:50921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:38489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=45205
--tserver_master_addrs=127.28.152.190:42877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:31.009236 9925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:31.011036 9925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:31.039952 9942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:31.040822 9941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:31.041272 9944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:31.042786 9925 server_base.cc:1034] running on GCE node
I20260926 02:16:31.043676 9925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:31.047091 9925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:31.048521 9925 hybrid_clock.cc:648] HybridClock initialized: now 1790388991048436 us; error 79 us; skew 500 ppm
I20260926 02:16:31.048965 9925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:31 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:31.053436 9925 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:16:31.057884 9925 webserver.cc:460] Webserver started at http://127.28.152.130:45205/ using document root <none> and password file <none>
I20260926 02:16:31.058934 9925 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:31.059113 9925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:16:31 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:16:31.095432 9925 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.005s sys 0.007s
I20260926 02:16:31.099726 9952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:31.101811 9925 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:16:31.102129 9925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "ffd364faeeb24c99917db63dc664902c"
format_stamp: "Formatted at 2026-09-26 02:16:27 on dist-test-slave-hllj"
server_key: "acc5f36dc3bbe80803ea8816ed1af122"
server_key_iv: "41a9f2df0e317b242b889807144eb48e"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:31.103678 9925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 02:16:31 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790388991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:31.137157 9925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:31.142964 9925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:31.143369 9925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:31.144719 9925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:31.146694 9925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:31.146847 9925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:16:31.146992 9925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:31.147094 9925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:31.191809 9925 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:38489
I20260926 02:16:31.191906 10065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:38489 every 8 connection(s)
I20260926 02:16:31.193903 9925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:16:31.198081 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 9925
I20260926 02:16:31.198863 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.152.148:50921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:33583
--local_ip_for_outbound_sockets=127.28.152.131
--tserver_master_addrs=127.28.152.190:42877
--webserver_port=40921
--webserver_interface=127.28.152.131
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.152.148:50921/kms with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:16:31 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:31.232282 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:31.197432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:43507 (local address 127.28.152.190:42877)
0926 02:16:31.214384 (+ 16952us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:31.214392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:31.214420 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:16:31.216616 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:31.216622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:31.216729 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:31.216839 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:31.218655 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:31.219378 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:31.221525 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:31.221967 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:31.222084 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:31.225998 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:31.226006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:31.226011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:31.226052 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:31.228806 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:31.229750 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:31.229755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:31.229758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:31.229922 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:31.230443 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:31.230446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:31.230449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:31.230762 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:31.230909 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:31.231684 (+ 775us) server_negotiation.cc:299] Negotiation successful
0926 02:16:31.231918 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16802,"thread_start_us":16667,"threads_started":1}
I20260926 02:16:31.233429 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:31.202297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:42877 (local address 127.28.152.130:43507)
0926 02:16:31.213211 (+ 10914us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:31.213288 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:31.216274 (+ 2986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:31.217283 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:31.217294 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:31.217784 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:31.218439 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:31.218459 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:31.219568 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:31.219572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:31.221347 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:31.221371 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:31.222239 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:31.222248 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:31.222477 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:31.223404 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:31.223473 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:31.225664 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:31.228974 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:31.228985 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:31.229006 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:31.229594 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:31.230067 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:31.230072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:31.230076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:31.230309 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:31.230896 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:31.230905 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:31.231285 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 02:16:31.231532 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 02:16:31.232113 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10466,"thread_start_us":10167,"threads_started":1}
I20260926 02:16:31.234957 10066 heartbeater.cc:347] Connected to a master server at 127.28.152.190:42877
I20260926 02:16:31.235280 10066 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:31.236104 10066 heartbeater.cc:511] Master 127.28.152.190:42877 requested a full tablet report, sending...
I20260926 02:16:31.237963 9724 ts_manager.cc:194] Registered new tserver with Master: ffd364faeeb24c99917db63dc664902c (127.28.152.130:38489)
I20260926 02:16:31.239705 9724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:43507
Sep 26 02:16:31 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:16:31.634120 9921 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:31.634595 9921 heartbeater.cc:511] Master 127.28.152.190:42877 requested a full tablet report, sending...
I20260926 02:16:31.635874 9724 ts_manager.cc:194] Registered new tserver with Master: 5a6709a5f1f14a968e988a8e5536bd41 (127.28.152.129:46135)
I20260926 02:16:31.637017 9724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:40199
W20260926 02:16:31.687171 10070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:31.687713 10070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:31.687934 10070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:31.699414 10070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:31.699644 10070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:31.699883 10070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:16:31.713574 10070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.152.148:50921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:33583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=40921
--tserver_master_addrs=127.28.152.190:42877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:31.715510 10070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:31.717324 10070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:31.770663 10076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:31.773834 10075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:31.774215 10079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:31.774830 10070 server_base.cc:1034] running on GCE node
I20260926 02:16:31.775818 10070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:31.786604 10070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:31.790138 10070 hybrid_clock.cc:648] HybridClock initialized: now 1790388991790031 us; error 100 us; skew 500 ppm
I20260926 02:16:31.790683 10070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:31 dist-test-slave-hllj krb5kdc[8705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:31.795156 10070 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:16:31.802827 10070 webserver.cc:460] Webserver started at http://127.28.152.131:40921/ using document root <none> and password file <none>
I20260926 02:16:31.803903 10070 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:31.804083 10070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:16:31 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:16:31.858506 10070 fs_manager.cc:709] Time spent opening directory manager: real 0.033s user 0.014s sys 0.009s
I20260926 02:16:31.873907 10086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:31.876327 10070 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.006s sys 0.009s
I20260926 02:16:31.876646 10070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "0d538681c5224242a96ef5b442d17edd"
format_stamp: "Formatted at 2026-09-26 02:16:28 on dist-test-slave-hllj"
server_key: "b43980a9918c0c61ac23717207e99e1c"
server_key_iv: "b7bb6010c34fa2084e59205db5a36ec0"
server_key_version: "kuduclusterkey@0"
I20260926 02:16:31.878441 10070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:31.954639 10070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:31.960340 10070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:31.960757 10070 kserver.cc:163] Server-wide thread pool size limit: 3276
Sep 26 02:16:31 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for HTTP/127.28.152.148@KRBTEST.COM
I20260926 02:16:31.981616 10070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:31.990389 10070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:31.990600 10070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:31.990772 10070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:31.990892 10070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:32.243654 10066 heartbeater.cc:503] Master 127.28.152.190:42877 was elected leader, sending a full tablet report...
I20260926 02:16:32.253000 10070 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:33583
I20260926 02:16:32.253054 10199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:33583 every 8 connection(s)
I20260926 02:16:32.254626 10070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:16:32.256143 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 10070
I20260926 02:16:32.305698 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.257358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.28.152.190:42877)
0926 02:16:32.284821 (+ 27463us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:32.284828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:32.284852 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:16:32.284920 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:32.284926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:32.285008 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:32.285121 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:32.287574 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.288743 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.291020 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.291738 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.291871 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.297694 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:32.297702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:32.297707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:32.297769 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:32.301400 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:32.302368 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:32.302374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:32.302378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:32.302547 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:32.303017 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:32.303022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:32.303025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:32.303389 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:32.303675 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:32.305051 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0926 02:16:32.305326 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27323,"thread_start_us":27151,"threads_started":1}
Sep 26 02:16:32 dist-test-slave-hllj krb5kdc[8705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:32.316023 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.284817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:42581 (local address 127.28.152.190:42877)
0926 02:16:32.295553 (+ 10736us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:32.295560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:32.295583 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:16:32.295646 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:32.295651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:32.295753 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:32.295891 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:32.298230 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.300140 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.302949 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.303614 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.303731 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.308597 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:32.308604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:32.308608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:32.308652 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:32.312300 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:32.313402 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:32.313408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:32.313412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:32.313554 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:32.314058 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:32.314063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:32.314066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:32.314436 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:32.314572 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:32.315406 (+ 834us) server_negotiation.cc:299] Negotiation successful
0926 02:16:32.315641 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10604,"thread_start_us":10477,"threads_started":1}
I20260926 02:16:32.317159 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.266005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:42877 (local address 127.28.152.131:42581)
0926 02:16:32.278849 (+ 12844us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:32.278928 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:32.283924 (+ 4996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:32.296234 (+ 12310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:32.296250 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:32.296932 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:32.297954 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.297987 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.300384 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.300390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.302770 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.302789 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.304045 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.304053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.304396 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.305263 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:32.305300 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:32.308287 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:32.312488 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:32.312500 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:32.312517 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:32.313261 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:32.313674 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:32.313680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:32.313684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:32.313913 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:32.314591 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:32.314602 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:32.314993 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 02:16:32.315296 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 02:16:32.315875 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12412,"thread_start_us":14991,"threads_started":1}
I20260926 02:16:32.318619 10200 heartbeater.cc:347] Connected to a master server at 127.28.152.190:42877
I20260926 02:16:32.318933 10200 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:32.319809 10200 heartbeater.cc:511] Master 127.28.152.190:42877 requested a full tablet report, sending...
I20260926 02:16:32.322190 9724 ts_manager.cc:194] Registered new tserver with Master: 0d538681c5224242a96ef5b442d17edd (127.28.152.131:33583)
I20260926 02:16:32.323357 9724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:42581
I20260926 02:16:32.328089 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:16:32.348353 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.332002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.28.152.190:42877)
0926 02:16:32.332217 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:32.332224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:32.332673 (+ 449us) server_negotiation.cc:407] Connection header received
0926 02:16:32.332877 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:32.332880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:32.332955 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:32.333037 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:32.335278 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.336079 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.338326 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.339018 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.339152 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.342860 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:32.342865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:32.342869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:32.342910 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:32.345549 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:32.346451 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:32.346455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:32.346458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:32.346586 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:32.347021 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:32.347024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:32.347026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:32.347324 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:32.347494 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:32.347927 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 02:16:32.348107 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:16:32.355027 9724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:16:32.366204 9724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:16:32.413841 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.382347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.28.152.130:38489)
0926 02:16:32.404317 (+ 21970us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:32.404328 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:32.404362 (+ 34us) server_negotiation.cc:407] Connection header received
0926 02:16:32.404478 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:32.404488 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:32.404811 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:32.405037 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:32.405783 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.407330 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.410398 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.412026 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.412221 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.412430 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:32.413359 (+ 929us) server_negotiation.cc:299] Negotiation successful
0926 02:16:32.413543 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21797,"thread_start_us":21658,"threads_started":1}
I20260926 02:16:32.413899 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.381888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:33583 (local address 127.0.0.1:48518)
0926 02:16:32.382885 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:32.382926 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:32.383092 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:32.406322 (+ 23230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:32.406326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:32.406353 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:32.406654 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.406665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.408730 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.408748 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.410883 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.410898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.412712 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.412718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.412908 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.413020 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 02:16:32.413175 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 02:16:32.413452 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":31,"thread_start_us":738,"threads_started":1}
I20260926 02:16:32.415241 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.382502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.28.152.131:33583)
0926 02:16:32.405675 (+ 23173us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:32.405683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:32.405712 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:16:32.405819 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:32.405826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:32.406098 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:32.406849 (+ 751us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:32.407112 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.408551 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.411079 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.412489 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.414472 (+ 1983us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.414674 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:32.414744 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 02:16:32.414916 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23000,"thread_start_us":22870,"threads_started":1}
I20260926 02:16:32.416126 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.382295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:38489 (local address 127.0.0.1:36784)
0926 02:16:32.403204 (+ 20909us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:32.403232 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:32.403420 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:32.405059 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:32.405066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:32.405098 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:32.405554 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.405568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.407489 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.407494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.410180 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.410198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.412599 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.412606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.412861 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.413022 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 02:16:32.413185 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 02:16:32.415749 (+ 2564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20756,"mutex_wait_us":1123,"thread_start_us":21392,"threads_started":1}
I20260926 02:16:32.418219 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.385959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46135 (local address 127.0.0.1:54830)
0926 02:16:32.403510 (+ 17551us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:32.403532 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:32.403628 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:32.409596 (+ 5968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:32.409606 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:32.409639 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:32.410146 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.410159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.413784 (+ 3625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.413790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.415470 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.415487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.417430 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.417437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.417619 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.417655 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:16:32.417731 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 02:16:32.417869 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17439,"mutex_wait_us":19,"thread_start_us":16936,"threads_started":1}
I20260926 02:16:32.419711 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.386056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.28.152.129:46135)
0926 02:16:32.408467 (+ 22411us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:32.408475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:32.408503 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:16:32.408597 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:32.408604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:32.408868 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:32.409084 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:32.410437 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.413613 (+ 3176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.415662 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.417122 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.418922 (+ 1800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.419110 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:32.419185 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 02:16:32.419364 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22235,"thread_start_us":23275,"threads_started":1}
I20260926 02:16:32.419926 10135 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6fbc5fb5714b20bb145a647a54c042 (DEFAULT_TABLE table=test-table [id=ff0fe38c0f354a8ba9b1f850fe4980ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:16:32.421846 10135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d6fbc5fb5714b20bb145a647a54c042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:32.421485 10000 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6fbc5fb5714b20bb145a647a54c042 (DEFAULT_TABLE table=test-table [id=ff0fe38c0f354a8ba9b1f850fe4980ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:16:32.425417 9856 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6fbc5fb5714b20bb145a647a54c042 (DEFAULT_TABLE table=test-table [id=ff0fe38c0f354a8ba9b1f850fe4980ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:16:32.426486 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d6fbc5fb5714b20bb145a647a54c042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:32.426918 9856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d6fbc5fb5714b20bb145a647a54c042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:32.444804 10228 tablet_bootstrap.cc:492] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: Bootstrap starting.
I20260926 02:16:32.447481 10229 tablet_bootstrap.cc:492] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41: Bootstrap starting.
I20260926 02:16:32.449803 10228 tablet_bootstrap.cc:654] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:32.451057 10229 tablet_bootstrap.cc:654] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:32.451176 10228 log.cc:824] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:32.452322 10229 log.cc:824] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:32.453542 10230 tablet_bootstrap.cc:492] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c: Bootstrap starting.
I20260926 02:16:32.456622 10228 tablet_bootstrap.cc:492] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: No bootstrap required, opened a new log
I20260926 02:16:32.456933 10228 ts_tablet_manager.cc:1397] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260926 02:16:32.457197 10229 tablet_bootstrap.cc:492] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41: No bootstrap required, opened a new log
I20260926 02:16:32.457558 10229 ts_tablet_manager.cc:1397] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:16:32.463281 10230 tablet_bootstrap.cc:654] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:32.464645 10230 log.cc:824] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:32.467566 10228 raft_consensus.cc:348] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d538681c5224242a96ef5b442d17edd" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 33583 } } peers { permanent_uuid: "ffd364faeeb24c99917db63dc664902c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 38489 } } peers { permanent_uuid: "5a6709a5f1f14a968e988a8e5536bd41" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46135 } }
I20260926 02:16:32.468123 10228 raft_consensus.cc:374] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:32.468250 10228 raft_consensus.cc:729] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d538681c5224242a96ef5b442d17edd, State: Initialized, Role: FOLLOWER
I20260926 02:16:32.469170 10228 consensus_queue.cc:260] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d538681c5224242a96ef5b442d17edd" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 33583 } } peers { permanent_uuid: "ffd364faeeb24c99917db63dc664902c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 38489 } } peers { permanent_uuid: "5a6709a5f1f14a968e988a8e5536bd41" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46135 } }
I20260926 02:16:32.470135 10229 raft_consensus.cc:348] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d538681c5224242a96ef5b442d17edd" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 33583 } } peers { permanent_uuid: "ffd364faeeb24c99917db63dc664902c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 38489 } } peers { permanent_uuid: "5a6709a5f1f14a968e988a8e5536bd41" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46135 } }
I20260926 02:16:32.470700 10229 raft_consensus.cc:374] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:32.470834 10229 raft_consensus.cc:729] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6709a5f1f14a968e988a8e5536bd41, State: Initialized, Role: FOLLOWER
I20260926 02:16:32.471632 10229 consensus_queue.cc:260] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d538681c5224242a96ef5b442d17edd" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 33583 } } peers { permanent_uuid: "ffd364faeeb24c99917db63dc664902c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 38489 } } peers { permanent_uuid: "5a6709a5f1f14a968e988a8e5536bd41" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46135 } }
I20260926 02:16:32.472400 10200 heartbeater.cc:503] Master 127.28.152.190:42877 was elected leader, sending a full tablet report...
I20260926 02:16:32.473507 10228 ts_tablet_manager.cc:1428] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260926 02:16:32.474488 10230 tablet_bootstrap.cc:492] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c: No bootstrap required, opened a new log
I20260926 02:16:32.474814 10230 ts_tablet_manager.cc:1397] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.006s
I20260926 02:16:32.477906 9921 heartbeater.cc:503] Master 127.28.152.190:42877 was elected leader, sending a full tablet report...
I20260926 02:16:32.478749 10229 ts_tablet_manager.cc:1428] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41: Time spent starting tablet: real 0.021s user 0.009s sys 0.009s
I20260926 02:16:32.484290 10230 raft_consensus.cc:348] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d538681c5224242a96ef5b442d17edd" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 33583 } } peers { permanent_uuid: "ffd364faeeb24c99917db63dc664902c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 38489 } } peers { permanent_uuid: "5a6709a5f1f14a968e988a8e5536bd41" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46135 } }
I20260926 02:16:32.484678 10230 raft_consensus.cc:374] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:32.484771 10230 raft_consensus.cc:729] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd364faeeb24c99917db63dc664902c, State: Initialized, Role: FOLLOWER
I20260926 02:16:32.485430 10230 consensus_queue.cc:260] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d538681c5224242a96ef5b442d17edd" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 33583 } } peers { permanent_uuid: "ffd364faeeb24c99917db63dc664902c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 38489 } } peers { permanent_uuid: "5a6709a5f1f14a968e988a8e5536bd41" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46135 } }
I20260926 02:16:32.487059 10230 ts_tablet_manager.cc:1428] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
W20260926 02:16:32.544010 10201 tablet.cc:2369] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:16:32.552419 9922 tablet.cc:2369] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:16:32.597292 10236 raft_consensus.cc:482] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:16:32.597651 10236 raft_consensus.cc:504] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d538681c5224242a96ef5b442d17edd" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 33583 } } peers { permanent_uuid: "ffd364faeeb24c99917db63dc664902c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 38489 } } peers { permanent_uuid: "5a6709a5f1f14a968e988a8e5536bd41" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46135 } }
I20260926 02:16:32.599428 10236 leader_election.cc:288] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d538681c5224242a96ef5b442d17edd (127.28.152.131:33583), 5a6709a5f1f14a968e988a8e5536bd41 (127.28.152.129:46135)
I20260926 02:16:32.607842 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.600574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:40765 (local address 127.28.152.129:46135)
0926 02:16:32.600797 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:32.600803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:32.601690 (+ 887us) server_negotiation.cc:407] Connection header received
0926 02:16:32.601835 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:32.601840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:32.601911 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:32.601992 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:32.602612 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.603657 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.605971 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.606804 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.606972 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.607055 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:32.607433 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 02:16:32.607586 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:16:32.608017 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.599933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:48771 (local address 127.28.152.131:33583)
0926 02:16:32.600152 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:32.600159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:32.601391 (+ 1232us) server_negotiation.cc:407] Connection header received
0926 02:16:32.601678 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:32.601684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:32.601759 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:32.601842 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:32.602474 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.603253 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.605850 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.607069 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.607234 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.607306 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:32.607640 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 02:16:32.607789 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 02:16:32.608306 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.600007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46135 (local address 127.28.152.130:40765)
0926 02:16:32.601560 (+ 1553us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:32.601577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:32.601711 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:32.602026 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:32.602034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:32.602059 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:32.602428 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.602439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.603808 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.603814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.605820 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.605830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.607036 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.607049 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.607230 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.607272 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:16:32.607378 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:16:32.607535 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"mutex_wait_us":3,"thread_start_us":827,"threads_started":1}
I20260926 02:16:32.608770 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.599919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:33583 (local address 127.28.152.130:48771)
0926 02:16:32.601253 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:32.601276 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:32.601513 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:32.601855 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:32.601859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:32.601879 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:32.602282 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.602296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.603403 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.603407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.605612 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:32.605631 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.607293 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.607298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:32.607460 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.607489 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 02:16:32.608280 (+ 791us) client_negotiation.cc:240] Negotiation successful
0926 02:16:32.608436 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":1,"thread_start_us":1324,"threads_started":1}
I20260926 02:16:32.609615 9876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6fbc5fb5714b20bb145a647a54c042" candidate_uuid: "ffd364faeeb24c99917db63dc664902c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6709a5f1f14a968e988a8e5536bd41" is_pre_election: true
I20260926 02:16:32.609766 10155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6fbc5fb5714b20bb145a647a54c042" candidate_uuid: "ffd364faeeb24c99917db63dc664902c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d538681c5224242a96ef5b442d17edd" is_pre_election: true
I20260926 02:16:32.610263 9876 raft_consensus.cc:2454] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd364faeeb24c99917db63dc664902c in term 0.
I20260926 02:16:32.610319 10155 raft_consensus.cc:2454] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd364faeeb24c99917db63dc664902c in term 0.
I20260926 02:16:32.611099 9954 leader_election.cc:302] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d538681c5224242a96ef5b442d17edd, ffd364faeeb24c99917db63dc664902c; no voters:
I20260926 02:16:32.611517 10236 raft_consensus.cc:2781] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:16:32.611677 10236 raft_consensus.cc:482] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:16:32.611786 10236 raft_consensus.cc:3037] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:32.614329 10236 raft_consensus.cc:504] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d538681c5224242a96ef5b442d17edd" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 33583 } } peers { permanent_uuid: "ffd364faeeb24c99917db63dc664902c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 38489 } } peers { permanent_uuid: "5a6709a5f1f14a968e988a8e5536bd41" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46135 } }
I20260926 02:16:32.614950 10236 leader_election.cc:288] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [CANDIDATE]: Term 1 election: Requested vote from peers 0d538681c5224242a96ef5b442d17edd (127.28.152.131:33583), 5a6709a5f1f14a968e988a8e5536bd41 (127.28.152.129:46135)
I20260926 02:16:32.615489 10155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6fbc5fb5714b20bb145a647a54c042" candidate_uuid: "ffd364faeeb24c99917db63dc664902c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d538681c5224242a96ef5b442d17edd"
I20260926 02:16:32.615530 9876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6fbc5fb5714b20bb145a647a54c042" candidate_uuid: "ffd364faeeb24c99917db63dc664902c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6709a5f1f14a968e988a8e5536bd41"
I20260926 02:16:32.615729 10155 raft_consensus.cc:3037] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:32.615742 9876 raft_consensus.cc:3037] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:32.619022 10155 raft_consensus.cc:2454] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd364faeeb24c99917db63dc664902c in term 1.
I20260926 02:16:32.619022 9876 raft_consensus.cc:2454] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd364faeeb24c99917db63dc664902c in term 1.
I20260926 02:16:32.619514 9954 leader_election.cc:302] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d538681c5224242a96ef5b442d17edd, ffd364faeeb24c99917db63dc664902c; no voters:
I20260926 02:16:32.619817 10236 raft_consensus.cc:2781] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:16:32.620338 10236 raft_consensus.cc:686] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 1 LEADER]: Becoming Leader. State: Replica: ffd364faeeb24c99917db63dc664902c, State: Running, Role: LEADER
I20260926 02:16:32.620900 10236 consensus_queue.cc:237] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d538681c5224242a96ef5b442d17edd" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 33583 } } peers { permanent_uuid: "ffd364faeeb24c99917db63dc664902c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 38489 } } peers { permanent_uuid: "5a6709a5f1f14a968e988a8e5536bd41" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46135 } }
I20260926 02:16:32.626308 9724 catalog_manager.cc:5495] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c reported cstate change: term changed from 0 to 1, leader changed from <none> to ffd364faeeb24c99917db63dc664902c (127.28.152.130). New cstate: current_term: 1 leader_uuid: "ffd364faeeb24c99917db63dc664902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d538681c5224242a96ef5b442d17edd" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 33583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffd364faeeb24c99917db63dc664902c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 38489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a6709a5f1f14a968e988a8e5536bd41" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46135 } health_report { overall_health: UNKNOWN } } }
I20260926 02:16:32.689567 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.683921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.28.152.130:38489)
0926 02:16:32.684133 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:32.684140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:32.684160 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:16:32.684337 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:32.684343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:32.684463 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:32.684625 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:16:32.685052 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.685840 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.687275 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.687700 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.687812 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.688359 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:16:32.688902 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:16:32.689070 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:32.689195 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 02:16:32.689293 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:16:32.698320 9876 raft_consensus.cc:1261] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41 [term 1 FOLLOWER]: Refusing update from remote peer ffd364faeeb24c99917db63dc664902c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:16:32.698323 10155 raft_consensus.cc:1261] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd [term 1 FOLLOWER]: Refusing update from remote peer ffd364faeeb24c99917db63dc664902c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:16:32.699642 10239 consensus_queue.cc:1035] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d538681c5224242a96ef5b442d17edd" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:16:32.700031 10236 consensus_queue.cc:1035] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a6709a5f1f14a968e988a8e5536bd41" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:16:32.712891 10242 mvcc.cc:205] Tried to move back new op lower bound from 7333433314081050624 to 7333433313779392512. Current Snapshot: MvccSnapshot[applied={T|T < 7333433314081050624}]
I20260926 02:16:32.739921 9723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:16:32.740211 9723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:16:32.744365 9723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e31d55439f24b3085a148e25cdcd779: Sending DeleteTablet for 3 replicas of tablet 4d6fbc5fb5714b20bb145a647a54c042
I20260926 02:16:32.745651 10000 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d6fbc5fb5714b20bb145a647a54c042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:16:32 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:36784
I20260926 02:16:32.745645 10135 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d6fbc5fb5714b20bb145a647a54c042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:16:32 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:48518
I20260926 02:16:32.746400 9856 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d6fbc5fb5714b20bb145a647a54c042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:16:32 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:54830
I20260926 02:16:32.749977 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 9761
I20260926 02:16:32.753633 10253 tablet_replica.cc:330] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: stopping tablet replica
I20260926 02:16:32.754143 10253 raft_consensus.cc:2229] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:16:32.754681 10253 raft_consensus.cc:2258] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:16:32.755663 10254 tablet_replica.cc:330] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41: stopping tablet replica
I20260926 02:16:32.756119 10254 raft_consensus.cc:2229] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:16:32.756670 10254 raft_consensus.cc:2258] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:16:32.757601 10252 tablet_replica.cc:330] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c: stopping tablet replica
I20260926 02:16:32.758023 10252 raft_consensus.cc:2229] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:16:32.758627 10252 raft_consensus.cc:2258] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:16:32.760269 10253 ts_tablet_manager.cc:1905] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:16:32.765120 10254 ts_tablet_manager.cc:1905] T 4d6fbc5fb5714b20bb145a647a54c042 P 5a6709a5f1f14a968e988a8e5536bd41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:16:32.772451 10252 ts_tablet_manager.cc:1905] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:16:32.772975 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.753656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.28.152.129:46135)
0926 02:16:32.754241 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:32.754247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:32.754266 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:16:32.754329 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:32.754334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:32.754412 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:32.754507 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:32.756414 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.757642 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.759598 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.760233 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.760364 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.764037 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:32.764056 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:32.764077 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:32.764126 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:32.768299 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:32.769451 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:32.769463 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:32.769467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:32.769658 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:32.770138 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:32.770143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:32.770146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:32.770634 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:32.770964 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:32.772298 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0926 02:16:32.772577 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434}
I20260926 02:16:32.773687 9916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:32.780260 10252 ts_tablet_manager.cc:1918] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:16:32.780486 10252 log.cc:1196] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/4d6fbc5fb5714b20bb145a647a54c042
I20260926 02:16:32.781085 10252 ts_tablet_manager.cc:1939] T 4d6fbc5fb5714b20bb145a647a54c042 P ffd364faeeb24c99917db63dc664902c: Deleting consensus metadata
I20260926 02:16:32.781307 10253 ts_tablet_manager.cc:1918] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:16:32.781548 10253 log.cc:1196] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/4d6fbc5fb5714b20bb145a647a54c042
I20260926 02:16:32.782123 10253 ts_tablet_manager.cc:1939] T 4d6fbc5fb5714b20bb145a647a54c042 P 0d538681c5224242a96ef5b442d17edd: Deleting consensus metadata
I20260926 02:16:32.783205 9708 catalog_manager.cc:4841] TS ffd364faeeb24c99917db63dc664902c (127.28.152.130:38489): tablet 4d6fbc5fb5714b20bb145a647a54c042 (table test-table [id=ff0fe38c0f354a8ba9b1f850fe4980ec]) successfully deleted
I20260926 02:16:32.784147 9709 catalog_manager.cc:4841] TS 0d538681c5224242a96ef5b442d17edd (127.28.152.131:33583): tablet 4d6fbc5fb5714b20bb145a647a54c042 (table test-table [id=ff0fe38c0f354a8ba9b1f850fe4980ec]) successfully deleted
I20260926 02:16:32.872380 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 9761
I20260926 02:16:32.886343 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 9925
W20260926 02:16:32.886394 9711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.152.129:46135 (error 108)
I20260926 02:16:32.888159 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.887778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46135 (local address 127.0.0.1:54850)
0926 02:16:32.887922 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:32.888021 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 02:16:32.888373 9711 catalog_manager.cc:4584] TS 5a6709a5f1f14a968e988a8e5536bd41 (127.28.152.129:46135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d6fbc5fb5714b20bb145a647a54c042: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46135: connect: Connection refused (error 111)
I20260926 02:16:32.903479 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.887130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.28.152.130:38489)
0926 02:16:32.887345 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:32.887351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:32.888182 (+ 831us) server_negotiation.cc:407] Connection header received
0926 02:16:32.888829 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:32.888834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:32.888902 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:32.888995 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:32.890721 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.891547 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.893456 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:32.893907 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:32.894006 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:32.897308 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:32.897323 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:32.897339 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:32.897385 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:32.900326 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:32.901347 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:32.901354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:32.901357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:32.901533 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:32.901951 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:32.901955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:32.901957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:32.902356 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:32.902610 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:32.902948 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 02:16:32.903140 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 02:16:32.904228 10061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:32.942281 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.941714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46135 (local address 127.0.0.1:54854)
0926 02:16:32.941950 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:32.942092 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 02:16:32.943681 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:32.943194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46135 (local address 127.0.0.1:54860)
0926 02:16:32.943400 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:32.943544 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 02:16:33.004374 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:33.003754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46135 (local address 127.0.0.1:54866)
0926 02:16:33.004033 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:33.004158 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260926 02:16:33.005662 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:33.005255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46135 (local address 127.0.0.1:54872)
0926 02:16:33.005410 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:33.005501 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 02:16:33.021075 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 9925
I20260926 02:16:33.037321 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 10070
I20260926 02:16:33.052237 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:33.038122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.28.152.131:33583)
0926 02:16:33.038317 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:33.038323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:33.038363 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:16:33.038569 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:33.038575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:33.038637 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:33.038748 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:33.040079 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:33.040816 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:33.042478 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:33.042932 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:33.043050 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:33.045907 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:33.045923 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:33.045942 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:33.045982 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:33.048989 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:33.049920 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:33.049930 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:33.049933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:33.050090 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:33.050539 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:33.050543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:33.050545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:33.050953 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:33.051280 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:33.051656 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 02:16:33.051885 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:16:33.052945 10195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:33.086686 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:33.086043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46135 (local address 127.0.0.1:54876)
0926 02:16:33.086348 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:33.086469 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260926 02:16:33.087926 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:33.087450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46135 (local address 127.0.0.1:54880)
0926 02:16:33.087676 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:33.087781 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 02:16:33.142148 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 10070
I20260926 02:16:33.155692 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 9691
I20260926 02:16:33.156935 9754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:33.267632 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 9691
I20260926 02:16:33.267655 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:33.267181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46135 (local address 127.0.0.1:54894)
0926 02:16:33.267370 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:33.267465 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 02:16:33.285914 29282 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 02:16:33.622170 29282 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 02:16:33.622402 29282 mini_ranger.cc:66] Stopping Ranger...
I20260926 02:16:33.958570 29282 mini_ranger.cc:68] Stopped Ranger
2026-09-26 02:16:33.958 UTC [8735] LOG: received smart shutdown request
2026-09-26 02:16:33.962 UTC [8735] LOG: background worker "logical replication launcher" (PID 8744) exited with exit code 1
2026-09-26 02:16:33.962 UTC [8739] LOG: shutting down
2026-09-26 02:16:34.140 UTC [8735] LOG: database system is shut down
2026-09-26T02:16:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54399 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:16:34 dist-test-slave-hllj krb5kdc[10268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:16:34 dist-test-slave-hllj krb5kdc[10268](info): set up 2 sockets
Sep 26 02:16:34 dist-test-slave-hllj krb5kdc[10268](info): commencing operation
krb5kdc: starting...
W20260926 02:16:35.258810 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:16:35 dist-test-slave-hllj krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:16:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:16:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:16:35.496968 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34027
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:35133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34027,127.28.152.189:45961,127.28.152.188:35389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:35.834071 10281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:35.834398 10281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:35.834489 10281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:35.841966 10281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:16:35.842092 10281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:35.842139 10281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:35.842196 10281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:16:35.842239 10281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:16:35.852404 10281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34027,127.28.152.189:45961,127.28.152.188:35389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:35.853826 10281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:35.855165 10281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:35.867376 10287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:35.867405 10286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:35.867682 10289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:35.868142 10281 server_base.cc:1034] running on GCE node
I20260926 02:16:35.868986 10281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:35.870808 10281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:35.872035 10281 hybrid_clock.cc:648] HybridClock initialized: now 1790388995871977 us; error 57 us; skew 500 ppm
I20260926 02:16:35.872459 10281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:35 dist-test-slave-hllj krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388995, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:35.880982 10281 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:16:35.882695 10281 webserver.cc:460] Webserver started at http://127.28.152.190:37573/ using document root <none> and password file <none>
I20260926 02:16:35.883471 10281 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:35.883589 10281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:35.883935 10281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:16:35.887181 10281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "efa315ad8734460a8618e9bf2363f7b6"
format_stamp: "Formatted at 2026-09-26 02:16:35 on dist-test-slave-hllj"
server_key: "ba14b1ab928d085cfe479ecb15e28650"
server_key_iv: "5da542c62e4c8f70d733e2b47d94b818"
server_key_version: "encryptionkey@0"
I20260926 02:16:35.888036 10281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "efa315ad8734460a8618e9bf2363f7b6"
format_stamp: "Formatted at 2026-09-26 02:16:35 on dist-test-slave-hllj"
server_key: "ba14b1ab928d085cfe479ecb15e28650"
server_key_iv: "5da542c62e4c8f70d733e2b47d94b818"
server_key_version: "encryptionkey@0"
I20260926 02:16:35.893785 10281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:16:35.897676 10296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:35.899435 10281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:16:35.899628 10281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "efa315ad8734460a8618e9bf2363f7b6"
format_stamp: "Formatted at 2026-09-26 02:16:35 on dist-test-slave-hllj"
server_key: "ba14b1ab928d085cfe479ecb15e28650"
server_key_iv: "5da542c62e4c8f70d733e2b47d94b818"
server_key_version: "encryptionkey@0"
I20260926 02:16:35.899853 10281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:35.922443 10281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:35.926942 10281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:35.927230 10281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:35.945693 10281 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:34027
I20260926 02:16:35.945716 10347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:34027 every 8 connection(s)
I20260926 02:16:35.947140 10281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:16:35.949972 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 10281
I20260926 02:16:35.950174 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:16:35.950806 29282 external_mini_cluster.cc:1428] Setting key 903e9b81b8a72276d46db4e13fc8ac7a
I20260926 02:16:35.951205 10348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:35.958221 10348 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.152.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.189@KRBTEST.COM" created.
Sep 26 02:16:35 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388995, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:35.984409 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:35.962656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.28.152.190:34027)
0926 02:16:35.964669 (+ 2013us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:35.964676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:35.964734 (+ 58us) server_negotiation.cc:407] Connection header received
0926 02:16:35.966734 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:35.966741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:35.967033 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:35.967227 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:35.968919 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:35.970253 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:35.972139 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:35.972734 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:35.972919 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:35.976357 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:35.976369 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:35.976388 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:35.976433 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:35.979855 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:35.980703 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:35.980712 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:35.980715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:35.980926 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:35.981412 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:35.981417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:35.981421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:35.982132 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:35.982382 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:35.982951 (+ 569us) server_negotiation.cc:299] Negotiation successful
0926 02:16:35.983221 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1792,"spinlock_wait_cycles":77184,"thread_start_us":1540,"threads_started":1}
I20260926 02:16:35.984472 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:35.962669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34027 (local address 127.0.0.1:37714)
0926 02:16:35.963646 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:35.963704 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:35.966252 (+ 2548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:35.967311 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:35.967320 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:35.968144 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:35.968675 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:35.968704 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:35.970434 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:35.970439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:35.972009 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:35.972022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:35.972959 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:35.972965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:35.973191 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:35.973901 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:35.973929 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:35.976100 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:35.980021 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:35.980033 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:35.980047 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:35.980562 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:35.981043 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:35.981047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:35.981050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:35.981293 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:35.982261 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:35.982267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:35.982536 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 02:16:35.982791 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 02:16:35.983417 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":91,"thread_start_us":423,"threads_started":1}
I20260926 02:16:35.987732 10348 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:16:35.988829 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:35.988385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:59930)
0926 02:16:35.988562 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:35.988690 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 02:16:35.989120 10300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
I20260926 02:16:35.990947 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:35.990485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:59938)
0926 02:16:35.990664 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:35.990758 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 02:16:35.991147 10348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:45961: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab.
Entry for principal kudu/127.28.152.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab.
I20260926 02:16:36.004043 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.189:45961
--webserver_interface=127.28.152.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab
--principal=kudu/127.28.152.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:35133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34027,127.28.152.189:45961,127.28.152.188:35389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:16:36.040556 10348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } attempt: 1
I20260926 02:16:36.041898 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.041347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:59954)
0926 02:16:36.041578 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.041731 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 02:16:36.043020 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.042592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:59958)
0926 02:16:36.042760 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.042850 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 02:16:36.043222 10348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:45961: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
I20260926 02:16:36.111545 10348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } attempt: 2
I20260926 02:16:36.112879 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.112308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:59964)
0926 02:16:36.112584 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.112701 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 02:16:36.114022 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.113583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:59976)
0926 02:16:36.113758 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.113847 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 02:16:36.114243 10348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:45961: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
I20260926 02:16:36.205541 10348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } attempt: 3
I20260926 02:16:36.206800 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.206218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:59984)
0926 02:16:36.206510 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.206635 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260926 02:16:36.207934 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.207475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:59988)
0926 02:16:36.207649 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.207748 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 02:16:36.208217 10348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:45961: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
W20260926 02:16:36.338910 10355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:36.339252 10355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:36.339484 10355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:36.347555 10355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:16:36.347720 10355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:36.347800 10355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:36.347877 10355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:16:36.347936 10355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:16:36.351480 10348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } attempt: 4
I20260926 02:16:36.352910 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.352276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:59998)
0926 02:16:36.352630 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.352747 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260926 02:16:36.354101 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.353656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:60008)
0926 02:16:36.353852 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.353957 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260926 02:16:36.354331 10348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:45961: Network error: Client connection negotiation failed: client connection to 127.28.152.189:45961: connect: Connection refused (error 111)
I20260926 02:16:36.360549 10355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34027,127.28.152.189:45961,127.28.152.188:35389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab
--principal=kudu/127.28.152.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.189:45961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.152.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:36.361969 10355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:36.363313 10355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:36.375701 10360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:36.375640 10363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:36.375623 10361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:36.377199 10355 server_base.cc:1034] running on GCE node
I20260926 02:16:36.378072 10355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:36.379628 10355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:36.380853 10355 hybrid_clock.cc:648] HybridClock initialized: now 1790388996380797 us; error 52 us; skew 500 ppm
I20260926 02:16:36.381271 10355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:36 dist-test-slave-hllj krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388996, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:36.384716 10355 init.cc:377] Logged in from keytab as kudu/127.28.152.189@KRBTEST.COM (short username kudu)
I20260926 02:16:36.386389 10355 webserver.cc:460] Webserver started at http://127.28.152.189:36367/ using document root <none> and password file <none>
I20260926 02:16:36.387172 10355 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:36.387315 10355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:36.387630 10355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:16:36.391042 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/data/instance:
uuid: "f3c12a02d008490d9d006cbdb1dcf16f"
format_stamp: "Formatted at 2026-09-26 02:16:36 on dist-test-slave-hllj"
server_key: "1baa6eeca2db4112370d9207fd604a7a"
server_key_iv: "c7e7a47b84d3240443b3e6c52a940f99"
server_key_version: "encryptionkey@0"
I20260926 02:16:36.391947 10355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/wal/instance:
uuid: "f3c12a02d008490d9d006cbdb1dcf16f"
format_stamp: "Formatted at 2026-09-26 02:16:36 on dist-test-slave-hllj"
server_key: "1baa6eeca2db4112370d9207fd604a7a"
server_key_iv: "c7e7a47b84d3240443b3e6c52a940f99"
server_key_version: "encryptionkey@0"
I20260926 02:16:36.397852 10355 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 02:16:36.401530 10370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:36.403026 10355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:16:36.403196 10355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/wal
uuid: "f3c12a02d008490d9d006cbdb1dcf16f"
format_stamp: "Formatted at 2026-09-26 02:16:36 on dist-test-slave-hllj"
server_key: "1baa6eeca2db4112370d9207fd604a7a"
server_key_iv: "c7e7a47b84d3240443b3e6c52a940f99"
server_key_version: "encryptionkey@0"
I20260926 02:16:36.403493 10355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:36.423633 10355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:36.427919 10355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:36.428194 10355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:36.446254 10355 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.189:45961
I20260926 02:16:36.446300 10421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.189:45961 every 8 connection(s)
I20260926 02:16:36.447723 10355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
I20260926 02:16:36.451398 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 10355
I20260926 02:16:36.451507 10422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:36.451587 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-1/wal/instance
I20260926 02:16:36.452041 29282 external_mini_cluster.cc:1428] Setting key 318044c688f16b381d27b82dd74a6050
I20260926 02:16:36.458573 10422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.152.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.188@KRBTEST.COM" created.
Sep 26 02:16:36 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388996, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:36.483247 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.461889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.28.152.190:34027)
0926 02:16:36.462103 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:36.462110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:36.463537 (+ 1427us) server_negotiation.cc:407] Connection header received
0926 02:16:36.465580 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:36.465586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:36.465676 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:36.465791 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:36.467743 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.468811 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.470587 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.471386 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.471542 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.475535 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:36.475543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:36.475548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:36.475593 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:36.478934 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.479953 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.479959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.479962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.480105 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.480568 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.480572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.480575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.480897 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:36.481014 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:36.482693 (+ 1679us) server_negotiation.cc:299] Negotiation successful
0926 02:16:36.482932 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:16:36.483960 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.462487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34027 (local address 127.0.0.1:37726)
0926 02:16:36.463348 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.463404 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:36.465299 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:36.466091 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:36.466101 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:36.466741 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:36.467503 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:36.467536 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.469008 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.469012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:36.470419 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:36.470432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.471638 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.471644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:36.471874 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.472577 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:36.472601 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:36.475236 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:36.479152 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:36.479163 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:36.479183 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:36.479797 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:36.480229 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:36.480234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:36.480237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:36.480459 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:36.481079 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:36.481087 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:36.481566 (+ 479us) client_negotiation.cc:769] Sending connection context
0926 02:16:36.481860 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 02:16:36.482466 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":361,"threads_started":1}
I20260926 02:16:36.485929 10422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:16:36 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388996, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:16:36.501852 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.486626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:60020)
0926 02:16:36.486943 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.486965 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:36.487078 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:36.488212 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:36.488217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:36.488681 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:36.488855 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:36.488866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.490330 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.490336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:36.491604 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:36.491621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.492575 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.492583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:36.492817 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.493463 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:36.493486 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:36.495847 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:36.499443 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:36.499448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:36.499452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:36.499801 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:36.500137 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:36.500140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:36.500143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:36.500321 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:36.500964 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:36.500968 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:36.501116 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 02:16:36.501276 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 02:16:36.501586 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 02:16:36.502200 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.486998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.28.152.189:45961)
0926 02:16:36.487764 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:36.487770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:36.487798 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:16:36.487843 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:36.487849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:36.488039 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:36.488156 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:36.489022 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.490187 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.491807 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.492407 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.492508 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.496207 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:36.496227 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:36.496244 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:36.496288 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:36.499303 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.499917 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.499924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.499927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.500045 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.500435 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.500440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.500442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.500827 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:36.501022 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:36.501424 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 02:16:36.501684 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":588,"thread_start_us":446,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab.
Entry for principal kudu/127.28.152.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab.
I20260926 02:16:36.504246 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.188:35389
--webserver_interface=127.28.152.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab
--principal=kudu/127.28.152.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:35133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34027,127.28.152.189:45961,127.28.152.188:35389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:16:36.504621 10422 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:16:36.505966 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.505503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58772)
0926 02:16:36.505693 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.505810 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 02:16:36.506282 10371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
I20260926 02:16:36.507982 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.507589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58782)
0926 02:16:36.507787 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.507873 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260926 02:16:36.508200 10422 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:35389: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
I20260926 02:16:36.557564 10422 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } attempt: 1
I20260926 02:16:36.558734 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.558279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58794)
0926 02:16:36.558478 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.558584 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 02:16:36.559868 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.559490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58806)
0926 02:16:36.559650 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.559726 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 02:16:36.560091 10422 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:35389: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
I20260926 02:16:36.628321 10422 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } attempt: 2
I20260926 02:16:36.629451 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.628955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58808)
0926 02:16:36.629192 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.629289 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 02:16:36.630396 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.630052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58820)
0926 02:16:36.630196 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.630270 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 02:16:36.630605 10422 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:35389: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
I20260926 02:16:36.653595 10348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } attempt: 5
Sep 26 02:16:36 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388995, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:16:36.669090 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.654290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:60036)
0926 02:16:36.654525 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.654547 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:36.654712 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:36.655152 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:36.655157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:36.655724 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:36.655996 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:36.656012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.657054 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.657059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:36.658400 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:36.658418 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.659253 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.659259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:36.659449 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.660094 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:36.660116 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:36.662587 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:36.666677 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:36.666682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:36.666685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:36.667072 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:36.667508 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:36.667512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:36.667514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:36.667671 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:36.668337 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:36.668340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:36.668482 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 02:16:36.668595 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 02:16:36.668845 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:16:36.669379 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.654422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.28.152.189:45961)
0926 02:16:36.654616 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:36.654621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:36.654640 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:16:36.654909 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:36.654913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:36.654986 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:36.655094 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:36.656191 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.656915 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.658569 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.659065 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.659189 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.662907 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:36.662915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:36.662922 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:36.662977 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:36.666490 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.667198 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.667204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.667207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.667367 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.667785 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.667790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.667793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.668195 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:36.668358 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:36.668752 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 02:16:36.669010 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 02:16:36.670449 10348 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:16:36.671646 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.671245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58834)
0926 02:16:36.671409 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.671492 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 02:16:36.672637 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.672283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58844)
0926 02:16:36.672461 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.672531 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 02:16:36.672842 10348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:35389: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
I20260926 02:16:36.721839 10422 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } attempt: 3
I20260926 02:16:36.723030 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.722558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58856)
0926 02:16:36.722757 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.722866 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 02:16:36.724071 10348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } attempt: 1
I20260926 02:16:36.724259 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.723679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58866)
0926 02:16:36.723905 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.724006 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260926 02:16:36.724588 10422 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:35389: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
I20260926 02:16:36.725057 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.724628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58878)
0926 02:16:36.724817 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.724922 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:16:36.726079 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.725741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58890)
0926 02:16:36.725897 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.725966 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 02:16:36.726279 10348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:35389: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
I20260926 02:16:36.794518 10348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } attempt: 2
I20260926 02:16:36.795732 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.795218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58898)
0926 02:16:36.795465 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.795555 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 02:16:36.796962 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.796572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58906)
0926 02:16:36.796748 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.796839 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 02:16:36.797227 10348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:35389: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
W20260926 02:16:36.844880 10429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:36.845286 10429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:36.845391 10429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:36.852983 10429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:16:36.853101 10429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:36.853257 10429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:36.853335 10429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:16:36.853376 10429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:16:36.863684 10429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34027,127.28.152.189:45961,127.28.152.188:35389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab
--principal=kudu/127.28.152.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.188:35389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.152.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:36.864962 10429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:36.866340 10429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 02:16:36.867811 10422 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } attempt: 4
I20260926 02:16:36.869217 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.868456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58912)
0926 02:16:36.868830 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.868968 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260926 02:16:36.870195 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.869848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58918)
0926 02:16:36.870007 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.870087 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 02:16:36.870407 10422 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:35389: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
W20260926 02:16:36.878204 10437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:36.878353 10434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:36.878311 10435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:36.879175 10429 server_base.cc:1034] running on GCE node
I20260926 02:16:36.880035 10429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:36.881520 10429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:36.882751 10429 hybrid_clock.cc:648] HybridClock initialized: now 1790388996882679 us; error 61 us; skew 500 ppm
I20260926 02:16:36.883129 10429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:36 dist-test-slave-hllj krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388996, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:36.886492 10429 init.cc:377] Logged in from keytab as kudu/127.28.152.188@KRBTEST.COM (short username kudu)
I20260926 02:16:36.888042 10429 webserver.cc:460] Webserver started at http://127.28.152.188:39387/ using document root <none> and password file <none>
I20260926 02:16:36.888751 10429 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:36.888866 10429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:36.889216 10429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:16:36.892336 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/data/instance:
uuid: "1836f927fb614f19b47114d7dd810d0a"
format_stamp: "Formatted at 2026-09-26 02:16:36 on dist-test-slave-hllj"
server_key: "160c7c5de4b42fc4605845128a37e24d"
server_key_iv: "3961c6dcab6a8ddb0b9e6364210fb5ba"
server_key_version: "encryptionkey@0"
I20260926 02:16:36.893229 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/wal/instance:
uuid: "1836f927fb614f19b47114d7dd810d0a"
format_stamp: "Formatted at 2026-09-26 02:16:36 on dist-test-slave-hllj"
server_key: "160c7c5de4b42fc4605845128a37e24d"
server_key_iv: "3961c6dcab6a8ddb0b9e6364210fb5ba"
server_key_version: "encryptionkey@0"
I20260926 02:16:36.898703 10429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 02:16:36.902447 10444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:36.903424 10348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } attempt: 3
I20260926 02:16:36.904075 10429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:16:36.904280 10429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/wal
uuid: "1836f927fb614f19b47114d7dd810d0a"
format_stamp: "Formatted at 2026-09-26 02:16:36 on dist-test-slave-hllj"
server_key: "160c7c5de4b42fc4605845128a37e24d"
server_key_iv: "3961c6dcab6a8ddb0b9e6364210fb5ba"
server_key_version: "encryptionkey@0"
I20260926 02:16:36.904505 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.903956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58926)
0926 02:16:36.904182 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.904293 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 02:16:36.904526 10429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:36.905623 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.905163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58936)
0926 02:16:36.905372 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.905477 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260926 02:16:36.905841 10348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:35389: Network error: Client connection negotiation failed: client connection to 127.28.152.188:35389: connect: Connection refused (error 111)
I20260926 02:16:36.930401 10429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:36.935379 10429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:36.935699 10429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:36.955125 10429 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.188:35389
I20260926 02:16:36.955147 10495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.188:35389 every 8 connection(s)
I20260926 02:16:36.956557 10429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
I20260926 02:16:36.957201 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 10429
I20260926 02:16:36.957434 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790388791636863-29282-0/minicluster-data/master-2/wal/instance
I20260926 02:16:36.957984 29282 external_mini_cluster.cc:1428] Setting key 3c265677ce9e05ee4a726f38a01dc867
I20260926 02:16:36.961066 10496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:36.968223 10496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:16:36 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
Sep 26 02:16:36 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388996, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:36.981604 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.959954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.28.152.188:35389)
0926 02:16:36.960755 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:36.960768 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:36.960818 (+ 50us) server_negotiation.cc:407] Connection header received
0926 02:16:36.962493 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:36.962517 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:36.962924 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:36.963300 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:36.964821 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.966272 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.967719 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.968463 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.968660 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.973481 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:36.973491 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:36.973505 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:36.973553 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:36.976993 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.977874 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.977882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.977892 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.978096 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.978546 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.978551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.978554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.979270 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:36.979500 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:36.980012 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 02:16:36.980382 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":372,"threads_started":1}
I20260926 02:16:36.986853 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.971110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.28.152.190:34027)
0926 02:16:36.971350 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:36.971357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:36.971976 (+ 619us) server_negotiation.cc:407] Connection header received
0926 02:16:36.972167 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:36.972171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:36.972236 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:36.972331 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:36.973599 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.974480 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.976501 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.976975 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.977096 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.980732 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:36.980737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:36.980740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:36.980779 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:36.983695 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.984541 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.984547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.984550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.984722 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.985194 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.985199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.985203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.985559 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:36.985662 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:36.986292 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 02:16:36.986499 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:16:36.986889 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.971043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34027 (local address 127.0.0.1:37736)
0926 02:16:36.971837 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.971876 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:36.972040 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:36.972425 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:36.972433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:36.973057 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:36.973448 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:36.973459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.974654 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.974660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:36.976320 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:36.976336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.977197 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.977202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:36.977334 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.977900 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:36.978003 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:36.980532 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:36.983842 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:36.983851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:36.983856 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:36.984417 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:36.984847 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:36.984852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:36.984856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:36.985030 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:36.985786 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:36.985792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:36.985953 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 02:16:36.986124 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 02:16:36.986467 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"spinlock_wait_cycles":22400,"thread_start_us":318,"threads_started":1}
I20260926 02:16:36.988596 10496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:16:36 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
Sep 26 02:16:36 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388996, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:16:37.000156 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.984993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.28.152.190:34027)
0926 02:16:36.985571 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:36.985575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:36.985593 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:16:36.985631 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:36.985634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:36.985685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:36.985772 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:36.987100 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.988009 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.989476 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.989908 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.990021 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.994300 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:36.994306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:36.994309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:36.994345 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:36.997268 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.998121 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.998126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.998128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.998254 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.998632 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.998636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.998638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.998945 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:36.999187 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:36.999549 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 02:16:36.999774 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":299,"threads_started":1}
I20260926 02:16:37.001565 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.989244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:45961 (local address 127.0.0.1:60040)
0926 02:16:36.989401 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:36.989421 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:36.989504 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:36.989913 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:36.989918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:36.990278 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:36.990420 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:36.990428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.991341 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.991347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:36.992238 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:36.992250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.993000 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.993005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:36.993245 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.993822 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:36.993840 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:36.995987 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:36.999042 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:36.999048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:36.999052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:36.999448 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:36.999844 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:36.999849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:36.999853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:37.000029 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:37.000622 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:37.000626 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:37.000774 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 02:16:37.000910 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 02:16:37.001227 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 02:16:37.001626 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:36.989452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.28.152.189:45961)
0926 02:16:36.989644 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:36.989649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:36.989663 (+ 14us) server_negotiation.cc:407] Connection header received
0926 02:16:36.989715 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:36.989718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:36.989774 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:36.989854 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:36.990563 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.991207 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.992381 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:36.992846 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:36.992957 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:36.996196 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:36.996203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:36.996208 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:36.996252 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:36.998896 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:36.999575 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:36.999579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:36.999581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:36.999728 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.000162 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.000168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.000171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.000498 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:37.000634 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:37.001024 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 02:16:37.001269 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:16:37.002863 10496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:16:37 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388996, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
Sep 26 02:16:37 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:16:37.015776 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:37.003584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58940)
0926 02:16:37.003736 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:37.003755 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:37.003965 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:37.004229 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:37.004233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:37.004691 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:37.004827 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:37.004836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.005817 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.005822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:37.006648 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:37.006658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.007558 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.007564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:37.007749 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:37.008272 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:37.008294 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:37.010595 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:37.013554 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:37.013559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:37.013564 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:37.013881 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:37.014212 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:37.014217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:37.014221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:37.014357 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:37.014817 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:37.014821 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:37.014951 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 02:16:37.015077 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 02:16:37.015379 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 02:16:37.016090 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:37.003779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.28.152.188:35389)
0926 02:16:37.003913 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:37.003917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:37.003929 (+ 12us) server_negotiation.cc:407] Connection header received
0926 02:16:37.004065 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:37.004070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:37.004128 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:37.004199 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:37.004952 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.005700 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.006829 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.007409 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.007518 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:37.010764 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:37.010769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:37.010772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:37.010808 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:37.013441 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.013987 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.013993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.013996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.014114 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.014449 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.014454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.014458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.014710 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:37.014841 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:37.015176 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 02:16:37.015544 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":42}
I20260926 02:16:37.016669 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:37.002010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.28.152.189:45961)
0926 02:16:37.002169 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:37.002174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:37.002194 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:16:37.002294 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:37.002299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:37.002365 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:37.002438 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:37.003872 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.004704 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.006077 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.006500 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.006622 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:37.011378 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:37.011384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:37.011387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:37.011421 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:37.014094 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.014792 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.014797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.014799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.014938 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.015314 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.015317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.015319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.015580 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:37.015836 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:37.016186 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 02:16:37.016432 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:16:37.022116 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:16:37.022248 10496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a: Bootstrap starting.
I20260926 02:16:37.026538 10496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:37.027956 10496 log.cc:824] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:37.038950 10496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a: No bootstrap required, opened a new log
I20260926 02:16:37.050640 10496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } }
I20260926 02:16:37.051337 10496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:37.051545 10496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1836f927fb614f19b47114d7dd810d0a, State: Initialized, Role: FOLLOWER
I20260926 02:16:37.051743 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:37.027895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.28.152.190:34027)
0926 02:16:37.028119 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:37.028126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:37.029900 (+ 1774us) server_negotiation.cc:407] Connection header received
0926 02:16:37.030282 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:37.030290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:37.030388 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:37.030476 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:37.033516 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.034714 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.037841 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.038533 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.038644 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:37.042790 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:37.042798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:37.042803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:37.042851 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:37.046781 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.047995 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.048002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.048006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.048163 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.049821 (+ 1658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.049826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.049829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.050231 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:37.050494 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:37.051131 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 02:16:37.051400 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:16:37.052829 10496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } }
I20260926 02:16:37.053614 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:37.030035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.28.152.189:45961)
0926 02:16:37.030224 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:37.030230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:37.036410 (+ 6180us) server_negotiation.cc:407] Connection header received
0926 02:16:37.036717 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:37.036723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:37.036822 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:37.036895 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:37.039103 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.040145 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.042350 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.042977 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.043076 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:37.046611 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:37.046618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:37.046623 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:37.046668 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:37.049641 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.050446 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.050452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.050456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.050625 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.051084 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.051089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.051092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.051544 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:37.051772 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:37.052968 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0926 02:16:37.053265 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:16:37.053637 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:37.028248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58956 (local address 127.28.152.188:35389)
0926 02:16:37.028470 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:37.028476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:37.035007 (+ 6531us) server_negotiation.cc:407] Connection header received
0926 02:16:37.035229 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:37.035236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:37.035349 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:37.035452 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:37.038590 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.039688 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.042062 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.042776 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.042891 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:37.046514 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:37.046521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:37.046526 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:37.046573 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:37.050193 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.051168 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.051173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.051176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.051347 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.051795 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.051813 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.051817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.052202 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:37.052442 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:37.053013 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 02:16:37.053287 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:16:37.055881 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } } }
I20260926 02:16:37.056247 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:16:37.056576 10496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:16:37.067701 10521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:16:37.067871 10521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:16:37.063665 10429 master_runner.cc:186] Error getting master registration for 127.28.152.190:34027: OK, instance_id { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" instance_seqno: 1790388995940452 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:16:37.073458 10429 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:16:37.083078 10348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } attempt: 4
Sep 26 02:16:37 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388995, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
I20260926 02:16:37.098037 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:37.083769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58968)
0926 02:16:37.083963 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:37.083983 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:37.084090 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:37.084498 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:37.084502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:37.084898 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:37.085098 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:37.085107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.086218 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.086223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:37.087589 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:37.087606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.088445 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.088452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:37.088630 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:37.089257 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:37.089275 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:37.091559 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:37.094670 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:37.094676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:37.094681 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:37.095074 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:37.095464 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:37.095469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:37.095473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:37.095608 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:37.096939 (+ 1331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:37.096944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:37.097171 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 02:16:37.097329 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 02:16:37.097638 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 02:16:37.098047 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:37.083992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.28.152.188:35389)
0926 02:16:37.084191 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:37.084197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:37.084214 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:16:37.084267 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:37.084272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:37.084363 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:37.084443 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:37.085275 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.086081 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.087747 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.088308 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.088418 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:37.091769 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:37.091774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:37.091777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:37.091822 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:37.094523 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.095200 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.095205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.095208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.095351 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.095714 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.095719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.095721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.096088 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:37.096871 (+ 783us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:37.097470 (+ 599us) server_negotiation.cc:299] Negotiation successful
0926 02:16:37.097704 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:16:37.104012 10348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6: Bootstrap starting.
I20260926 02:16:37.108438 10348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:37.109918 10348 log.cc:824] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:37.113397 10348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6: No bootstrap required, opened a new log
I20260926 02:16:37.124415 10348 raft_consensus.cc:348] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } }
I20260926 02:16:37.124871 10348 raft_consensus.cc:374] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:37.125000 10348 raft_consensus.cc:729] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa315ad8734460a8618e9bf2363f7b6, State: Initialized, Role: FOLLOWER
I20260926 02:16:37.125751 10348 consensus_queue.cc:260] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } }
I20260926 02:16:37.126711 10524 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } } }
I20260926 02:16:37.127038 10524 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:16:37.128186 10348 sys_catalog.cc:564] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:16:37.137491 10535 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:16:37.137627 10535 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:16:37.143033 10524 raft_consensus.cc:482] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:16:37.143356 10524 raft_consensus.cc:504] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } }
I20260926 02:16:37.145195 10524 leader_election.cc:288] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c12a02d008490d9d006cbdb1dcf16f (127.28.152.189:45961), 1836f927fb614f19b47114d7dd810d0a (127.28.152.188:35389)
I20260926 02:16:37.146135 10471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efa315ad8734460a8618e9bf2363f7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1836f927fb614f19b47114d7dd810d0a" is_pre_election: true
I20260926 02:16:37.146765 10471 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa315ad8734460a8618e9bf2363f7b6 in term 0.
I20260926 02:16:37.147398 10297 leader_election.cc:302] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1836f927fb614f19b47114d7dd810d0a, efa315ad8734460a8618e9bf2363f7b6; no voters:
I20260926 02:16:37.147868 10524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:16:37.148011 10524 raft_consensus.cc:482] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:16:37.148114 10524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:37.151609 10524 raft_consensus.cc:504] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } }
I20260926 02:16:37.152987 10471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efa315ad8734460a8618e9bf2363f7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1836f927fb614f19b47114d7dd810d0a"
I20260926 02:16:37.153283 10471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:37.156790 10471 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa315ad8734460a8618e9bf2363f7b6 in term 1.
I20260926 02:16:37.157259 10297 leader_election.cc:302] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1836f927fb614f19b47114d7dd810d0a, efa315ad8734460a8618e9bf2363f7b6; no voters:
I20260926 02:16:37.158474 10537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:16:37.158676 10524 leader_election.cc:288] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [CANDIDATE]: Term 1 election: Requested vote from peers f3c12a02d008490d9d006cbdb1dcf16f (127.28.152.189:45961), 1836f927fb614f19b47114d7dd810d0a (127.28.152.188:35389)
I20260926 02:16:37.159019 10537 raft_consensus.cc:686] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [term 1 LEADER]: Becoming Leader. State: Replica: efa315ad8734460a8618e9bf2363f7b6, State: Running, Role: LEADER
I20260926 02:16:37.160604 10537 consensus_queue.cc:237] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } }
I20260926 02:16:37.139089 10281 master_runner.cc:186] Error getting master registration for 127.28.152.189:45961: OK, instance_id { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" instance_seqno: 1790388996440721 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:16:37.163645 10281 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:16:37.167977 10524 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efa315ad8734460a8618e9bf2363f7b6. Latest consensus state: current_term: 1 leader_uuid: "efa315ad8734460a8618e9bf2363f7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } } }
I20260926 02:16:37.168294 10524 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:37.169337 10539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:16:37.169593 10422 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } attempt: 5
Sep 26 02:16:37 dist-test-slave-hllj krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790388996, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
W20260926 02:16:37.180400 10302 tablet.cc:2369] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:16:37.180732 10539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:16:37.188050 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:37.170248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:35389 (local address 127.0.0.1:58970)
0926 02:16:37.170661 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:37.170683 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:37.170796 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:37.171711 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:37.171716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:37.172221 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:37.172446 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:37.172458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.173897 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.173903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:37.175235 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:37.175251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.176435 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.176441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:37.176649 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:37.177331 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:37.177354 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:37.180248 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:37.184987 (+ 4739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:37.184992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:37.184996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:37.185550 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:37.185992 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:37.185997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:37.186001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:37.186206 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:37.186949 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:37.186955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:37.187129 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 02:16:37.187359 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 02:16:37.187671 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278}
W20260926 02:16:37.188046 10450 tablet.cc:2369] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:16:37.189236 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:37.171099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.28.152.188:35389)
0926 02:16:37.171271 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:37.171275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:37.171292 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:16:37.171350 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:37.171355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:37.171428 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:37.171530 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:37.172622 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.173726 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.175419 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:37.176094 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:37.176210 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:37.180471 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:37.180478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:37.180483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:37.180525 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:37.184814 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.185696 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.185703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.185707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.185856 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:37.186343 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:37.186348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:37.186351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:37.186719 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:37.186887 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:37.188535 (+ 1648us) server_negotiation.cc:299] Negotiation successful
0926 02:16:37.188797 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:16:37.195688 10422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f: Bootstrap starting.
I20260926 02:16:37.204083 10422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:37.205819 10422 log.cc:824] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:37.220880 10471 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [term 1 FOLLOWER]: Refusing update from remote peer efa315ad8734460a8618e9bf2363f7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:16:37.223654 10524 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:16:37.237848 10422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f: No bootstrap required, opened a new log
I20260926 02:16:37.254123 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader efa315ad8734460a8618e9bf2363f7b6. Latest consensus state: current_term: 1 leader_uuid: "efa315ad8734460a8618e9bf2363f7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } } }
I20260926 02:16:37.254480 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:16:37.256765 10540 mvcc.cc:205] Tried to move back new op lower bound from 7333433332582785024 to 7333433332389847040. Current Snapshot: MvccSnapshot[applied={T|T < 7333433332582785024}]
I20260926 02:16:37.260253 10542 mvcc.cc:205] Tried to move back new op lower bound from 7333433332582785024 to 7333433332389847040. Current Snapshot: MvccSnapshot[applied={T|T < 7333433332582785024}]
I20260926 02:16:37.260828 10422 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } }
I20260926 02:16:37.261276 10422 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:37.261417 10422 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c12a02d008490d9d006cbdb1dcf16f, State: Initialized, Role: FOLLOWER
I20260926 02:16:37.262176 10422 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } }
I20260926 02:16:37.266381 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efa315ad8734460a8618e9bf2363f7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } } }
I20260926 02:16:37.266654 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:16:37.269265 10546 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } } }
I20260926 02:16:37.269703 10546 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:16:37.271603 10422 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:16:37.271659 10396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efa315ad8734460a8618e9bf2363f7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c12a02d008490d9d006cbdb1dcf16f"
I20260926 02:16:37.272157 10396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:37.274315 10397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efa315ad8734460a8618e9bf2363f7b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" is_pre_election: true
I20260926 02:16:37.278868 10396 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa315ad8734460a8618e9bf2363f7b6 in term 1.
I20260926 02:16:37.279630 10395 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [term 1 FOLLOWER]: Refusing update from remote peer efa315ad8734460a8618e9bf2363f7b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:16:37.281544 10537 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:16:37.286628 10539 catalog_manager.cc:1347] Generated new cluster ID: ab575b23bb16465aacee5ae1a7fb3e9e
I20260926 02:16:37.286756 10539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:16:37.287240 10537 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efa315ad8734460a8618e9bf2363f7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } } }
I20260926 02:16:37.287555 10537 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:37.314003 10539 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:16:37.316053 10539 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:16:37.333360 10539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6: Generated new TSK 0
I20260926 02:16:37.334689 10539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:16:37.337747 10546 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [sys.catalog]: SysCatalogTable state changed. Reason: New leader efa315ad8734460a8618e9bf2363f7b6. Latest consensus state: current_term: 1 leader_uuid: "efa315ad8734460a8618e9bf2363f7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } } }
I20260926 02:16:37.338119 10546 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:16:37.338544 10552 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efa315ad8734460a8618e9bf2363f7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } } }
I20260926 02:16:37.338944 10552 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:16:37.340335 10524 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efa315ad8734460a8618e9bf2363f7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa315ad8734460a8618e9bf2363f7b6" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34027 } } peers { permanent_uuid: "f3c12a02d008490d9d006cbdb1dcf16f" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 45961 } } peers { permanent_uuid: "1836f927fb614f19b47114d7dd810d0a" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 35389 } } }
I20260926 02:16:37.340643 10524 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa315ad8734460a8618e9bf2363f7b6 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:37.396122 10565 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:16:38.070024 10521 catalog_manager.cc:1259] Loaded cluster ID: ab575b23bb16465aacee5ae1a7fb3e9e
I20260926 02:16:38.070130 10521 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a: loading cluster ID for follower catalog manager: success
I20260926 02:16:38.072647 10521 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a: acquiring CA information for follower catalog manager: success
I20260926 02:16:38.074525 10521 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1836f927fb614f19b47114d7dd810d0a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:16:38.400955 10565 catalog_manager.cc:1259] Loaded cluster ID: ab575b23bb16465aacee5ae1a7fb3e9e
I20260926 02:16:38.401072 10565 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f: loading cluster ID for follower catalog manager: success
I20260926 02:16:38.403851 10565 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f: acquiring CA information for follower catalog manager: success
I20260926 02:16:38.406083 10565 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f3c12a02d008490d9d006cbdb1dcf16f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:16:38.486080 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 10281
I20260926 02:16:38.488175 10343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:38.602356 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 10281
I20260926 02:16:38.620560 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 10355
I20260926 02:16:38.621748 10417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:38.729962 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 10355
I20260926 02:16:38.744834 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 10429
I20260926 02:16:38.745981 10491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:38.852685 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 10429
2026-09-26T02:16:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4609 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:16:38 dist-test-slave-hllj krb5kdc[10573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:16:38 dist-test-slave-hllj krb5kdc[10573](info): set up 2 sockets
Sep 26 02:16:38 dist-test-slave-hllj krb5kdc[10573](info): commencing operation
krb5kdc: starting...
W20260926 02:16:39.870733 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:16:40 dist-test-slave-hllj krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:16:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:16:40.101914 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34881
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:40.447073 10586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:40.447389 10586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:40.447510 10586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:40.455066 10586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:16:40.455190 10586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:40.455241 10586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:40.455327 10586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:16:40.455384 10586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:16:40.465667 10586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:40.467032 10586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:40.468506 10586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:40.480569 10591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:40.480523 10594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:40.480485 10592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:40.481411 10586 server_base.cc:1034] running on GCE node
I20260926 02:16:40.482187 10586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:40.483712 10586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:40.484930 10586 hybrid_clock.cc:648] HybridClock initialized: now 1790389000484878 us; error 39 us; skew 500 ppm
I20260926 02:16:40.485335 10586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:40 dist-test-slave-hllj krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389000, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:40.488418 10586 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:16:40.490118 10586 webserver.cc:460] Webserver started at http://127.28.152.190:40475/ using document root <none> and password file <none>
I20260926 02:16:40.490832 10586 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:40.490952 10586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:40.491255 10586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:16:40.494436 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "14682b27e4114734beb10e9f8450db51"
format_stamp: "Formatted at 2026-09-26 02:16:40 on dist-test-slave-hllj"
server_key: "66a406e91a3301b0bfed214dd5f8ae52"
server_key_iv: "5645c2c3705581260c7bc8630c09cdfe"
server_key_version: "encryptionkey@0"
I20260926 02:16:40.495183 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "14682b27e4114734beb10e9f8450db51"
format_stamp: "Formatted at 2026-09-26 02:16:40 on dist-test-slave-hllj"
server_key: "66a406e91a3301b0bfed214dd5f8ae52"
server_key_iv: "5645c2c3705581260c7bc8630c09cdfe"
server_key_version: "encryptionkey@0"
I20260926 02:16:40.500545 10586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:16:40.504230 10601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:40.505709 10586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:16:40.505889 10586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "14682b27e4114734beb10e9f8450db51"
format_stamp: "Formatted at 2026-09-26 02:16:40 on dist-test-slave-hllj"
server_key: "66a406e91a3301b0bfed214dd5f8ae52"
server_key_iv: "5645c2c3705581260c7bc8630c09cdfe"
server_key_version: "encryptionkey@0"
I20260926 02:16:40.506083 10586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:40.524268 10586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:40.528764 10586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:40.529040 10586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:40.547405 10586 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:34881
I20260926 02:16:40.547416 10652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:34881 every 8 connection(s)
I20260926 02:16:40.548861 10586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:16:40.552901 10653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:40.554857 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 10586
I20260926 02:16:40.555088 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:16:40.555732 29282 external_mini_cluster.cc:1428] Setting key 4c8e2cc330192b9a95c70b67ffd28478
I20260926 02:16:40.565214 10653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51: Bootstrap starting.
I20260926 02:16:40.568390 10653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:40.569471 10653 log.cc:824] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:16:40 dist-test-slave-hllj krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:40.572587 10653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51: No bootstrap required, opened a new log
I20260926 02:16:40.579582 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:40.557755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.28.152.190:34881)
0926 02:16:40.558626 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:40.558640 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:40.558696 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:16:40.560470 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:40.560496 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:40.560951 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:40.561401 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:40.563045 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:40.564426 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:40.565985 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:40.566713 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:40.566895 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:40.571389 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:40.571398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:40.571416 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:40.571455 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:40.574729 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:40.575765 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:40.575773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:40.575781 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:40.575987 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:40.576451 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:40.576455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:40.576457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:40.577088 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:40.577411 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:40.577868 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 02:16:40.578387 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":379,"threads_started":1}
I20260926 02:16:40.581557 10653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14682b27e4114734beb10e9f8450db51" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34881 } }
I20260926 02:16:40.581945 10653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:40.582070 10653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14682b27e4114734beb10e9f8450db51, State: Initialized, Role: FOLLOWER
I20260926 02:16:40.582679 10653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14682b27e4114734beb10e9f8450db51" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34881 } }
I20260926 02:16:40.582924 10653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:16:40.583074 10653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:16:40.583252 10653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:40.583982 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:16:40.585691 10653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14682b27e4114734beb10e9f8450db51" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34881 } }
I20260926 02:16:40.586292 10653 leader_election.cc:302] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14682b27e4114734beb10e9f8450db51; no voters:
I20260926 02:16:40.587024 10653 leader_election.cc:288] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:16:40.587121 10659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:16:40.588055 10659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [term 1 LEADER]: Becoming Leader. State: Replica: 14682b27e4114734beb10e9f8450db51, State: Running, Role: LEADER
I20260926 02:16:40.588761 10659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14682b27e4114734beb10e9f8450db51" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34881 } }
I20260926 02:16:40.589406 10653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:16:40.591944 10661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14682b27e4114734beb10e9f8450db51. Latest consensus state: current_term: 1 leader_uuid: "14682b27e4114734beb10e9f8450db51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14682b27e4114734beb10e9f8450db51" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34881 } } }
I20260926 02:16:40.591940 10660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14682b27e4114734beb10e9f8450db51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14682b27e4114734beb10e9f8450db51" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34881 } } }
I20260926 02:16:40.592474 10661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:40.592571 10660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:40.595243 10668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:16:40.599774 10668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:16:40.609225 10668 catalog_manager.cc:1347] Generated new cluster ID: d44a17111736434cb92d1eaa8e7ae293
I20260926 02:16:40.609388 10668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:16:40.618268 10668 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:16:40.620267 10668 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:16:40.628373 10668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14682b27e4114734beb10e9f8450db51: Generated new TSK 0
I20260926 02:16:40.629695 10668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:16:40.644608 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 10586
I20260926 02:16:40.645776 10648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:40.735904 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 10586
2026-09-26T02:16:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1880 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:16:40 dist-test-slave-hllj krb5kdc[10682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:16:40 dist-test-slave-hllj krb5kdc[10682](info): set up 2 sockets
Sep 26 02:16:40 dist-test-slave-hllj krb5kdc[10682](info): commencing operation
krb5kdc: starting...
W20260926 02:16:41.762864 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:16:41 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:16:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:16:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:16:42.004786 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:45565
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:42.337970 10695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:42.338281 10695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:42.338392 10695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:42.345845 10695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:16:42.345963 10695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:42.346029 10695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:42.346084 10695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:16:42.346128 10695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:16:42.356328 10695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:42.357750 10695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:42.359077 10695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:42.371013 10700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:42.370913 10703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:42.370951 10701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:42.371971 10695 server_base.cc:1034] running on GCE node
I20260926 02:16:42.372882 10695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:42.374936 10695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:42.376160 10695 hybrid_clock.cc:648] HybridClock initialized: now 1790389002376103 us; error 47 us; skew 500 ppm
I20260926 02:16:42.376531 10695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:42 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389002, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:42.393687 10695 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:16:42.395368 10695 webserver.cc:460] Webserver started at http://127.28.152.190:38427/ using document root <none> and password file <none>
I20260926 02:16:42.396144 10695 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:42.396274 10695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:42.396562 10695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:16:42.399837 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "3f23e98402ae4d208b61e430bc77e262"
format_stamp: "Formatted at 2026-09-26 02:16:42 on dist-test-slave-hllj"
server_key: "1e2e37a55372f855e19641d7b59b95ac"
server_key_iv: "9dbc4d2239d06af1047bcbb0afe6f9a0"
server_key_version: "encryptionkey@0"
I20260926 02:16:42.400676 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "3f23e98402ae4d208b61e430bc77e262"
format_stamp: "Formatted at 2026-09-26 02:16:42 on dist-test-slave-hllj"
server_key: "1e2e37a55372f855e19641d7b59b95ac"
server_key_iv: "9dbc4d2239d06af1047bcbb0afe6f9a0"
server_key_version: "encryptionkey@0"
I20260926 02:16:42.406826 10695 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 02:16:42.410670 10710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:42.412178 10695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:16:42.412351 10695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "3f23e98402ae4d208b61e430bc77e262"
format_stamp: "Formatted at 2026-09-26 02:16:42 on dist-test-slave-hllj"
server_key: "1e2e37a55372f855e19641d7b59b95ac"
server_key_iv: "9dbc4d2239d06af1047bcbb0afe6f9a0"
server_key_version: "encryptionkey@0"
I20260926 02:16:42.412549 10695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:42.428932 10695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:42.433593 10695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:42.433877 10695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:42.451504 10695 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:45565
I20260926 02:16:42.451555 10762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:45565 every 8 connection(s)
I20260926 02:16:42.452895 10695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:16:42.456789 10763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:42.457613 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 10695
I20260926 02:16:42.457844 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:16:42.458671 29282 external_mini_cluster.cc:1428] Setting key 34041d8f7958d27fcbbc6bfd9fb1bf86
I20260926 02:16:42.472107 10763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Bootstrap starting.
Sep 26 02:16:42 dist-test-slave-hllj krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:42.475559 10763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:42.476938 10763 log.cc:824] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:42.480520 10763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: No bootstrap required, opened a new log
I20260926 02:16:42.484687 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:42.460892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.28.152.190:45565)
0926 02:16:42.461725 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:42.461742 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:42.461798 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:16:42.463980 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:42.464014 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:42.464640 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:42.465182 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:42.466583 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:42.468613 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:42.470154 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:42.471252 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:42.471437 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:42.475675 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:42.475688 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:42.475709 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:42.475764 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:42.479597 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:42.480434 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:42.480443 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:42.480452 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:42.480729 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:42.481166 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:42.481173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:42.481176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:42.481935 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:42.482260 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:42.482663 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 02:16:42.483134 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":349,"threads_started":1}
I20260926 02:16:42.490170 10763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:42.490540 10763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:42.490640 10763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f23e98402ae4d208b61e430bc77e262, State: Initialized, Role: FOLLOWER
I20260926 02:16:42.491297 10763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:42.491573 10763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:16:42.491706 10763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:16:42.491864 10763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:42.494199 10763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:42.494745 10763 leader_election.cc:302] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f23e98402ae4d208b61e430bc77e262; no voters:
I20260926 02:16:42.495498 10763 leader_election.cc:288] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:16:42.495609 10768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:16:42.496208 10768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 1 LEADER]: Becoming Leader. State: Replica: 3f23e98402ae4d208b61e430bc77e262, State: Running, Role: LEADER
I20260926 02:16:42.496868 10768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:42.497833 10763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:16:42.499815 10769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f23e98402ae4d208b61e430bc77e262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } } }
I20260926 02:16:42.500213 10769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:42.501839 10777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:16:42.501962 10770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f23e98402ae4d208b61e430bc77e262. Latest consensus state: current_term: 1 leader_uuid: "3f23e98402ae4d208b61e430bc77e262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } } }
I20260926 02:16:42.502274 10770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:42.507691 10777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:16:42.517388 10777 catalog_manager.cc:1347] Generated new cluster ID: 1d277b4275a34c1fa2e3d3cc0fcdcd7b
I20260926 02:16:42.517521 10777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:16:42.564040 10777 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:16:42.566289 10777 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:16:42.576386 10777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Generated new TSK 0
I20260926 02:16:42.577415 10777 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:16:42.633086 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45565
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:42.965281 10789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:42.965605 10789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:42.965730 10789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:42.973201 10789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:42.973323 10789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:42.973479 10789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:16:42.983659 10789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:42.985029 10789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:42.986408 10789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:42.999846 10795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:42.999907 10797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:43.000458 10794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:43.000715 10789 server_base.cc:1034] running on GCE node
I20260926 02:16:43.001380 10789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:43.002380 10789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:43.003597 10789 hybrid_clock.cc:648] HybridClock initialized: now 1790389003003542 us; error 39 us; skew 500 ppm
I20260926 02:16:43.003929 10789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:43 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:43.007107 10789 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:16:43.008566 10789 webserver.cc:460] Webserver started at http://127.28.152.129:45047/ using document root <none> and password file <none>
I20260926 02:16:43.009425 10789 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:43.009558 10789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:43.009868 10789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:16:43.012938 10789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "ceb55118b8574e85950eb94585993b12"
format_stamp: "Formatted at 2026-09-26 02:16:43 on dist-test-slave-hllj"
server_key: "b24678412101106165e71829be238e59"
server_key_iv: "f6ad04d2f3a499c2fda5ed80a0177fee"
server_key_version: "encryptionkey@0"
I20260926 02:16:43.013815 10789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "ceb55118b8574e85950eb94585993b12"
format_stamp: "Formatted at 2026-09-26 02:16:43 on dist-test-slave-hllj"
server_key: "b24678412101106165e71829be238e59"
server_key_iv: "f6ad04d2f3a499c2fda5ed80a0177fee"
server_key_version: "encryptionkey@0"
I20260926 02:16:43.019505 10789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:16:43.023394 10804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:43.024823 10789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:16:43.024993 10789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "ceb55118b8574e85950eb94585993b12"
format_stamp: "Formatted at 2026-09-26 02:16:43 on dist-test-slave-hllj"
server_key: "b24678412101106165e71829be238e59"
server_key_iv: "f6ad04d2f3a499c2fda5ed80a0177fee"
server_key_version: "encryptionkey@0"
I20260926 02:16:43.025231 10789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:43.054152 10789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:43.058902 10789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:43.059191 10789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:43.060231 10789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:43.061654 10789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:43.061728 10789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:43.061843 10789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:43.061901 10789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:43.092346 10789 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:46827
I20260926 02:16:43.092386 10918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:46827 every 8 connection(s)
I20260926 02:16:43.093737 10789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:16:43.096299 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 10789
I20260926 02:16:43.096519 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:16:43.097215 29282 external_mini_cluster.cc:1428] Setting key 986c526b0b2b3a4b4fcd32039409a473
Sep 26 02:16:43 dist-test-slave-hllj krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:43.115425 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:43.096299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:34487 (local address 127.28.152.190:45565)
0926 02:16:43.096845 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:43.096852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:43.098016 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 02:16:43.100294 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:43.100300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:43.100370 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:43.100457 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:43.102515 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:43.103249 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:43.105466 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:43.105890 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:43.106005 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:43.109912 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:43.109917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:43.109920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:43.109974 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:43.112387 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:43.113261 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:43.113265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:43.113268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:43.113383 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:43.113812 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:43.113815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:43.113817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:43.114084 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:43.114191 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:43.114992 (+ 801us) server_negotiation.cc:299] Negotiation successful
0926 02:16:43.115165 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":276,"threads_started":1}
I20260926 02:16:43.116850 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:43.096902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:34487)
0926 02:16:43.097754 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:43.097804 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:43.099997 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:43.100737 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:43.100749 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:43.101527 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:43.102319 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:43.102344 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:43.103407 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:43.103412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:43.105316 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:43.105330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:43.106107 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:43.106113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:43.106399 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:43.107107 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:43.107135 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:43.109660 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:43.112533 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:43.112543 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:43.112560 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:43.113158 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:43.113468 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:43.113472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:43.113475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:43.113718 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:43.114193 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:43.114201 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:43.114626 (+ 425us) client_negotiation.cc:769] Sending connection context
0926 02:16:43.114898 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 02:16:43.115450 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":329,"threads_started":1}
I20260926 02:16:43.118480 10919 heartbeater.cc:347] Connected to a master server at 127.28.152.190:45565
I20260926 02:16:43.118772 10919 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:43.119495 10919 heartbeater.cc:511] Master 127.28.152.190:45565 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
I20260926 02:16:43.121544 10727 ts_manager.cc:194] Registered new tserver with Master: ceb55118b8574e85950eb94585993b12 (127.28.152.129:46827)
I20260926 02:16:43.123054 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:34487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:16:43.151957 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45565
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:43.480261 10925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:43.480569 10925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:43.480671 10925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:43.488117 10925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:43.488248 10925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:43.488380 10925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:16:43.498889 10925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:43.500371 10925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:43.501777 10925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:43.515213 10930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:43.515249 10933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:43.515283 10931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:43.516173 10925 server_base.cc:1034] running on GCE node
I20260926 02:16:43.516861 10925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:43.517998 10925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:43.519218 10925 hybrid_clock.cc:648] HybridClock initialized: now 1790389003519168 us; error 53 us; skew 500 ppm
I20260926 02:16:43.519557 10925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:43 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:43.522754 10925 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:16:43.524258 10925 webserver.cc:460] Webserver started at http://127.28.152.130:46309/ using document root <none> and password file <none>
I20260926 02:16:43.524956 10925 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:43.525077 10925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:43.525404 10925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:16:43.528398 10925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "83e3270277b9431bb9e7afa69fa0b69c"
format_stamp: "Formatted at 2026-09-26 02:16:43 on dist-test-slave-hllj"
server_key: "6e3ea97c84889375759ac787736273d4"
server_key_iv: "b918b8e98ca2c60db65ca8c6e37f0378"
server_key_version: "encryptionkey@0"
I20260926 02:16:43.529299 10925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "83e3270277b9431bb9e7afa69fa0b69c"
format_stamp: "Formatted at 2026-09-26 02:16:43 on dist-test-slave-hllj"
server_key: "6e3ea97c84889375759ac787736273d4"
server_key_iv: "b918b8e98ca2c60db65ca8c6e37f0378"
server_key_version: "encryptionkey@0"
I20260926 02:16:43.534981 10925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:16:43.538655 10940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:43.540088 10925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:16:43.540252 10925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "83e3270277b9431bb9e7afa69fa0b69c"
format_stamp: "Formatted at 2026-09-26 02:16:43 on dist-test-slave-hllj"
server_key: "6e3ea97c84889375759ac787736273d4"
server_key_iv: "b918b8e98ca2c60db65ca8c6e37f0378"
server_key_version: "encryptionkey@0"
I20260926 02:16:43.540441 10925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:43.558791 10925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:43.562933 10925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:43.563222 10925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:43.564211 10925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:43.565716 10925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:43.565795 10925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:43.565932 10925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:43.565986 10925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:43.596836 10925 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:45943
I20260926 02:16:43.596864 11054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:45943 every 8 connection(s)
I20260926 02:16:43.598253 10925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:16:43.603710 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 10925
I20260926 02:16:43.603897 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:16:43.604313 29282 external_mini_cluster.cc:1428] Setting key 44148356aea2b95f5fb0edad594859fe
Sep 26 02:16:43 dist-test-slave-hllj krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:43.618516 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:43.600944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:33717 (local address 127.28.152.190:45565)
0926 02:16:43.601210 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:43.601215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:43.602541 (+ 1326us) server_negotiation.cc:407] Connection header received
0926 02:16:43.604496 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:43.604502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:43.604596 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:43.604686 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:43.606353 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:43.607057 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:43.608667 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:43.609081 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:43.609230 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:43.612756 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:43.612761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:43.612765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:43.612801 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:43.615267 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:43.616017 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:43.616023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:43.616026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:43.616215 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:43.616634 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:43.616640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:43.616643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:43.616918 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:43.617049 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:43.617928 (+ 879us) server_negotiation.cc:299] Negotiation successful
0926 02:16:43.618168 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 02:16:43.619558 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:43.601605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:33717)
0926 02:16:43.602368 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:43.602409 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:43.604182 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:43.604962 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:43.604971 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:43.605555 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:43.606182 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:43.606201 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:43.607208 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:43.607212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:43.608522 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:43.608535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:43.609352 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:43.609360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:43.609606 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:43.610263 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:43.610292 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:43.612508 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:43.615410 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:43.615420 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:43.615438 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:43.615897 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:43.616322 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:43.616327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:43.616331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:43.616531 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:43.617045 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:43.617053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:43.617548 (+ 495us) client_negotiation.cc:769] Sending connection context
0926 02:16:43.617836 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 02:16:43.618419 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":317,"threads_started":1}
I20260926 02:16:43.620904 11055 heartbeater.cc:347] Connected to a master server at 127.28.152.190:45565
I20260926 02:16:43.621233 11055 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:43.621986 11055 heartbeater.cc:511] Master 127.28.152.190:45565 requested a full tablet report, sending...
I20260926 02:16:43.623435 10727 ts_manager.cc:194] Registered new tserver with Master: 83e3270277b9431bb9e7afa69fa0b69c (127.28.152.130:45943)
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:16:43.624296 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:33717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:16:43.652673 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45565
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:43.986537 11060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:43.986837 11060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:43.986932 11060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:43.994341 11060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:43.994467 11060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:43.994622 11060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:16:44.004822 11060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:44.006194 11060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:44.007550 11060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:44.021159 11068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:44.021237 11066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:44.021215 11065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:44.021890 11060 server_base.cc:1034] running on GCE node
I20260926 02:16:44.022576 11060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:44.023573 11060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:44.024834 11060 hybrid_clock.cc:648] HybridClock initialized: now 1790389004024766 us; error 49 us; skew 500 ppm
I20260926 02:16:44.025204 11060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:44 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389004, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:44.028549 11060 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:16:44.030288 11060 webserver.cc:460] Webserver started at http://127.28.152.131:36979/ using document root <none> and password file <none>
I20260926 02:16:44.030995 11060 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:44.031098 11060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:44.031374 11060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:16:44.034544 11060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "a8fe6fe7f98c4205894dca7f9d8b1107"
format_stamp: "Formatted at 2026-09-26 02:16:44 on dist-test-slave-hllj"
server_key: "9bd76286cf567925de5e176ac25ee91a"
server_key_iv: "edca37cddb90db8d1228561db370ca58"
server_key_version: "encryptionkey@0"
I20260926 02:16:44.035302 11060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "a8fe6fe7f98c4205894dca7f9d8b1107"
format_stamp: "Formatted at 2026-09-26 02:16:44 on dist-test-slave-hllj"
server_key: "9bd76286cf567925de5e176ac25ee91a"
server_key_iv: "edca37cddb90db8d1228561db370ca58"
server_key_version: "encryptionkey@0"
I20260926 02:16:44.040766 11060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 02:16:44.044466 11075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:44.046078 11060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:16:44.046244 11060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "a8fe6fe7f98c4205894dca7f9d8b1107"
format_stamp: "Formatted at 2026-09-26 02:16:44 on dist-test-slave-hllj"
server_key: "9bd76286cf567925de5e176ac25ee91a"
server_key_iv: "edca37cddb90db8d1228561db370ca58"
server_key_version: "encryptionkey@0"
I20260926 02:16:44.046444 11060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:44.087666 11060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:44.092140 11060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:44.092446 11060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:44.093575 11060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:44.095021 11060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:44.095096 11060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:44.095216 11060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:44.095270 11060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:44.126200 10919 heartbeater.cc:503] Master 127.28.152.190:45565 was elected leader, sending a full tablet report...
I20260926 02:16:44.126444 11060 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:35381
I20260926 02:16:44.126477 11189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:35381 every 8 connection(s)
I20260926 02:16:44.127992 11060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:16:44.135896 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 11060
I20260926 02:16:44.136060 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:16:44.136528 29282 external_mini_cluster.cc:1428] Setting key b1fd48ace57c530ff4743d40e874c330
Sep 26 02:16:44 dist-test-slave-hllj krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389004, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:44.147516 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.130612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:59607 (local address 127.28.152.190:45565)
0926 02:16:44.131067 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:44.131072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:44.131941 (+ 869us) server_negotiation.cc:407] Connection header received
0926 02:16:44.133802 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:44.133808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:44.133895 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:44.133992 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:44.135495 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.136339 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.138100 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.138602 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.138711 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.141987 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:44.141992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:44.141996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:44.142036 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:44.144651 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:44.145504 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:44.145508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:44.145511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:44.145644 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:44.146000 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:44.146003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:44.146005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:44.146271 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:44.146387 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:44.147052 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 02:16:44.147233 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":251,"threads_started":1}
I20260926 02:16:44.148527 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.131067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:59607)
0926 02:16:44.131760 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:44.131808 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:44.133531 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:44.134232 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:44.134240 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:44.134745 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:44.135329 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.135345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.136486 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.136491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.137970 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.137982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.138811 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.138818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.139048 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.139695 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:44.139717 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:44.141790 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:44.144773 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:44.144781 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:44.144796 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:44.145347 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:44.145735 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:44.145738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:44.145741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:44.145917 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:44.146373 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:44.146379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:44.146723 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 02:16:44.146953 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 02:16:44.147373 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":277,"threads_started":1}
I20260926 02:16:44.150031 11190 heartbeater.cc:347] Connected to a master server at 127.28.152.190:45565
I20260926 02:16:44.150364 11190 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:44.151280 11190 heartbeater.cc:511] Master 127.28.152.190:45565 requested a full tablet report, sending...
I20260926 02:16:44.152655 10727 ts_manager.cc:194] Registered new tserver with Master: a8fe6fe7f98c4205894dca7f9d8b1107 (127.28.152.131:35381)
I20260926 02:16:44.153519 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:59607
I20260926 02:16:44.164104 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:16:44.180877 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.168571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.28.152.190:45565)
0926 02:16:44.168726 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:44.168731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:44.169066 (+ 335us) server_negotiation.cc:407] Connection header received
0926 02:16:44.169251 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:44.169254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:44.169376 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:44.169472 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:44.170801 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.171537 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.172986 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.173511 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.173624 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.175998 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:44.176003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:44.176007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:44.176042 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:44.178539 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:44.179171 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:44.179175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:44.179178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:44.179297 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:44.179604 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:44.179607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:44.179609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:44.179930 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:44.180117 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:44.180456 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 02:16:44.180638 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 02:16:44.185000 10727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:16:44.191921 10727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:16:44.216188 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.204673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:35381 (local address 127.0.0.1:54862)
0926 02:16:44.205650 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:44.205708 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:44.205912 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:44.208471 (+ 2559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:44.208481 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:44.208522 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:44.209078 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.209094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.211143 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.211147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.212748 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.212762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.215007 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.215013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.215363 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.215443 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 02:16:44.215570 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 02:16:44.215737 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":74,"thread_start_us":484,"threads_started":1}
I20260926 02:16:44.217154 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.204984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.28.152.131:35381)
0926 02:16:44.206743 (+ 1759us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:44.206754 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:44.206791 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:16:44.206903 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:44.207858 (+ 955us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:44.208198 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:44.208422 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:44.209363 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.211015 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.212901 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.214426 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.214635 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.214847 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:44.216493 (+ 1646us) server_negotiation.cc:299] Negotiation successful
0926 02:16:44.216753 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1551,"thread_start_us":1371,"threads_started":1}
I20260926 02:16:44.218605 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.204851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:45943 (local address 127.0.0.1:35432)
0926 02:16:44.209180 (+ 4329us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:44.209202 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:44.209436 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:44.210331 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:44.210336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:44.210360 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:44.210698 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.210708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.212581 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.212587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.215358 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.215374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.217693 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.217701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.217879 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.217912 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:16:44.218012 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:16:44.218167 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4153,"mutex_wait_us":105,"thread_start_us":4238,"threads_started":1}
I20260926 02:16:44.218619 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.205223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.28.152.130:45943)
0926 02:16:44.206205 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:44.206215 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:44.209304 (+ 3089us) server_negotiation.cc:407] Connection header received
0926 02:16:44.209635 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:44.209644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:44.209882 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:44.210062 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:44.210869 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.212403 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.215603 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.217164 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.217337 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.217539 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:44.218106 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 02:16:44.218281 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":731,"threads_started":1}
I20260926 02:16:44.219411 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.207226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.28.152.129:46827)
0926 02:16:44.208078 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:44.208089 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:44.209698 (+ 1609us) server_negotiation.cc:407] Connection header received
0926 02:16:44.209795 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:44.209803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:44.210086 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:44.210260 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:44.211145 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.212627 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.215453 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.217529 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.217726 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.217904 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:44.218920 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0926 02:16:44.219099 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":412,"threads_started":1}
I20260926 02:16:44.219817 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.207052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46827 (local address 127.0.0.1:48150)
0926 02:16:44.209436 (+ 2384us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:44.209455 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:44.209572 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:44.210507 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:44.210512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:44.210537 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:44.210960 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.210973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.212788 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.212796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.215277 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.215292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.218046 (+ 2754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.218052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.218214 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.218245 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:16:44.218331 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 02:16:44.218458 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2237,"mutex_wait_us":47,"thread_start_us":1764,"threads_started":1}
I20260926 02:16:44.221308 11125 tablet_service.cc:1467] Processing CreateTablet for tablet e919bebcf3844b5ca16a30128a2552e4 (DEFAULT_TABLE table=test-table [id=fc456df28c634230acf6dabeb00af3a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:16:44.222873 11125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e919bebcf3844b5ca16a30128a2552e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:44.222589 10990 tablet_service.cc:1467] Processing CreateTablet for tablet e919bebcf3844b5ca16a30128a2552e4 (DEFAULT_TABLE table=test-table [id=fc456df28c634230acf6dabeb00af3a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:16:44.223193 10853 tablet_service.cc:1467] Processing CreateTablet for tablet e919bebcf3844b5ca16a30128a2552e4 (DEFAULT_TABLE table=test-table [id=fc456df28c634230acf6dabeb00af3a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:16:44.223982 10990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e919bebcf3844b5ca16a30128a2552e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:44.224340 10853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e919bebcf3844b5ca16a30128a2552e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:44.237840 11209 tablet_bootstrap.cc:492] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: Bootstrap starting.
I20260926 02:16:44.238530 11210 tablet_bootstrap.cc:492] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: Bootstrap starting.
I20260926 02:16:44.241853 11210 tablet_bootstrap.cc:654] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:44.241861 11209 tablet_bootstrap.cc:654] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:44.241904 11211 tablet_bootstrap.cc:492] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: Bootstrap starting.
I20260926 02:16:44.243247 11210 log.cc:824] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:44.243247 11209 log.cc:824] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:44.245569 11211 tablet_bootstrap.cc:654] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:44.246896 11211 log.cc:824] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:44.246978 11210 tablet_bootstrap.cc:492] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: No bootstrap required, opened a new log
I20260926 02:16:44.247262 11210 ts_tablet_manager.cc:1397] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:16:44.247303 11209 tablet_bootstrap.cc:492] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: No bootstrap required, opened a new log
I20260926 02:16:44.247535 11209 ts_tablet_manager.cc:1397] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260926 02:16:44.252033 11211 tablet_bootstrap.cc:492] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: No bootstrap required, opened a new log
I20260926 02:16:44.252333 11211 ts_tablet_manager.cc:1397] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260926 02:16:44.255743 11209 raft_consensus.cc:348] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } }
I20260926 02:16:44.256139 11209 raft_consensus.cc:374] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:44.256232 11209 raft_consensus.cc:729] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb55118b8574e85950eb94585993b12, State: Initialized, Role: FOLLOWER
I20260926 02:16:44.256956 11209 consensus_queue.cc:260] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } }
I20260926 02:16:44.257452 11210 raft_consensus.cc:348] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } }
I20260926 02:16:44.257915 11210 raft_consensus.cc:374] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:44.258021 11210 raft_consensus.cc:729] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fe6fe7f98c4205894dca7f9d8b1107, State: Initialized, Role: FOLLOWER
I20260926 02:16:44.258715 11210 consensus_queue.cc:260] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } }
I20260926 02:16:44.259955 11190 heartbeater.cc:503] Master 127.28.152.190:45565 was elected leader, sending a full tablet report...
I20260926 02:16:44.260628 11210 ts_tablet_manager.cc:1428] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260926 02:16:44.262360 11211 raft_consensus.cc:348] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } }
I20260926 02:16:44.262730 11211 raft_consensus.cc:374] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:44.262831 11211 raft_consensus.cc:729] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e3270277b9431bb9e7afa69fa0b69c, State: Initialized, Role: FOLLOWER
I20260926 02:16:44.263383 11211 consensus_queue.cc:260] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } }
I20260926 02:16:44.266371 11211 ts_tablet_manager.cc:1428] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260926 02:16:44.266739 11055 heartbeater.cc:503] Master 127.28.152.190:45565 was elected leader, sending a full tablet report...
I20260926 02:16:44.268066 11209 ts_tablet_manager.cc:1428] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: Time spent starting tablet: real 0.020s user 0.005s sys 0.012s
W20260926 02:16:44.346081 10920 tablet.cc:2369] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:16:44.350301 11056 tablet.cc:2369] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:16:44.379715 11191 tablet.cc:2369] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:16:44.390266 11216 raft_consensus.cc:482] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:16:44.390584 11216 raft_consensus.cc:504] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } }
I20260926 02:16:44.392282 11216 leader_election.cc:288] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e3270277b9431bb9e7afa69fa0b69c (127.28.152.130:45943), ceb55118b8574e85950eb94585993b12 (127.28.152.129:46827)
I20260926 02:16:44.398996 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.392834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:34765 (local address 127.28.152.129:46827)
0926 02:16:44.393047 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:44.393051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:44.393243 (+ 192us) server_negotiation.cc:407] Connection header received
0926 02:16:44.393482 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:44.393488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:44.393571 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:44.393666 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:44.394124 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.395081 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.397183 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.397997 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.398144 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.398222 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:44.398618 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 02:16:44.398746 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 02:16:44.399006 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.392755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:49199 (local address 127.28.152.130:45943)
0926 02:16:44.392971 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:44.392977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:44.393668 (+ 691us) server_negotiation.cc:407] Connection header received
0926 02:16:44.393841 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:44.393846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:44.393919 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:44.394024 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:44.394641 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.395394 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.397232 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.398009 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.398147 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.398222 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:44.398620 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 02:16:44.398757 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:16:44.399056 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.392728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46827 (local address 127.28.152.131:34765)
0926 02:16:44.393097 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:44.393150 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:44.393326 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:44.393739 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:44.393742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:44.393771 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:44.393955 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.393966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.395215 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.395220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.396976 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.396988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.398145 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.398173 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.398453 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.398490 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:16:44.398573 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 02:16:44.398715 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":162}
I20260926 02:16:44.399158 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.392742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:45943 (local address 127.28.152.131:49199)
0926 02:16:44.393541 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:44.393563 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:44.393697 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:44.394074 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:44.394078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:44.394106 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:44.394474 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.394487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.395543 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.395547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.397056 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:44.397070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.398217 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.398223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:44.398455 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.398494 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:16:44.398574 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 02:16:44.398718 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":117,"thread_start_us":422,"threads_started":1}
I20260926 02:16:44.400158 10873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e919bebcf3844b5ca16a30128a2552e4" candidate_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb55118b8574e85950eb94585993b12" is_pre_election: true
I20260926 02:16:44.400231 11010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e919bebcf3844b5ca16a30128a2552e4" candidate_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e3270277b9431bb9e7afa69fa0b69c" is_pre_election: true
I20260926 02:16:44.400664 10873 raft_consensus.cc:2454] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fe6fe7f98c4205894dca7f9d8b1107 in term 0.
I20260926 02:16:44.400678 11010 raft_consensus.cc:2454] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fe6fe7f98c4205894dca7f9d8b1107 in term 0.
I20260926 02:16:44.401331 11078 leader_election.cc:302] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83e3270277b9431bb9e7afa69fa0b69c, a8fe6fe7f98c4205894dca7f9d8b1107; no voters:
I20260926 02:16:44.401748 11216 raft_consensus.cc:2781] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:16:44.401907 11216 raft_consensus.cc:482] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:16:44.402020 11216 raft_consensus.cc:3037] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:44.404253 11216 raft_consensus.cc:504] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } }
I20260926 02:16:44.404964 11216 leader_election.cc:288] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [CANDIDATE]: Term 1 election: Requested vote from peers 83e3270277b9431bb9e7afa69fa0b69c (127.28.152.130:45943), ceb55118b8574e85950eb94585993b12 (127.28.152.129:46827)
I20260926 02:16:44.405516 11010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e919bebcf3844b5ca16a30128a2552e4" candidate_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e3270277b9431bb9e7afa69fa0b69c"
I20260926 02:16:44.405743 11010 raft_consensus.cc:3037] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:44.405735 10873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e919bebcf3844b5ca16a30128a2552e4" candidate_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb55118b8574e85950eb94585993b12"
I20260926 02:16:44.405937 10873 raft_consensus.cc:3037] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:44.408066 11010 raft_consensus.cc:2454] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fe6fe7f98c4205894dca7f9d8b1107 in term 1.
I20260926 02:16:44.408205 10873 raft_consensus.cc:2454] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fe6fe7f98c4205894dca7f9d8b1107 in term 1.
I20260926 02:16:44.408653 11079 leader_election.cc:302] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83e3270277b9431bb9e7afa69fa0b69c, a8fe6fe7f98c4205894dca7f9d8b1107, ceb55118b8574e85950eb94585993b12; no voters:
I20260926 02:16:44.409026 11216 raft_consensus.cc:2781] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:16:44.409516 11216 raft_consensus.cc:686] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 1 LEADER]: Becoming Leader. State: Replica: a8fe6fe7f98c4205894dca7f9d8b1107, State: Running, Role: LEADER
I20260926 02:16:44.410010 11216 consensus_queue.cc:237] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } }
I20260926 02:16:44.415012 10727 catalog_manager.cc:5495] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8fe6fe7f98c4205894dca7f9d8b1107 (127.28.152.131). New cstate: current_term: 1 leader_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } health_report { overall_health: UNKNOWN } } }
I20260926 02:16:44.466228 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.459920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.28.152.131:35381)
0926 02:16:44.460143 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:44.460150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:44.460226 (+ 76us) server_negotiation.cc:407] Connection header received
0926 02:16:44.460463 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:44.460467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:44.460559 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:44.460700 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:16:44.461079 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.461930 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.463950 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.464381 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.464497 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.465037 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:16:44.465656 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:16:44.465794 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:44.465885 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 02:16:44.465974 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:16:44.474568 10873 raft_consensus.cc:1261] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12 [term 1 FOLLOWER]: Refusing update from remote peer a8fe6fe7f98c4205894dca7f9d8b1107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:16:44.474568 11010 raft_consensus.cc:1261] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c [term 1 FOLLOWER]: Refusing update from remote peer a8fe6fe7f98c4205894dca7f9d8b1107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:16:44.475570 11219 consensus_queue.cc:1035] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:16:44.475926 11216 consensus_queue.cc:1035] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:16:44.486447 11222 mvcc.cc:205] Tried to move back new op lower bound from 7333433362317606912 to 7333433362067025920. Current Snapshot: MvccSnapshot[applied={T|T < 7333433362317606912}]
I20260926 02:16:44.500964 10727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:16:44.501264 10727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:16:44.505350 10727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Sending DeleteTablet for 3 replicas of tablet e919bebcf3844b5ca16a30128a2552e4
I20260926 02:16:44.506608 10990 tablet_service.cc:1514] Processing DeleteTablet for tablet e919bebcf3844b5ca16a30128a2552e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:16:44 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:35432
I20260926 02:16:44.506624 11125 tablet_service.cc:1514] Processing DeleteTablet for tablet e919bebcf3844b5ca16a30128a2552e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:16:44 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:54862
I20260926 02:16:44.507436 10853 tablet_service.cc:1514] Processing DeleteTablet for tablet e919bebcf3844b5ca16a30128a2552e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:16:44 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:48150
I20260926 02:16:44.507699 11233 tablet_replica.cc:330] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: stopping tablet replica
I20260926 02:16:44.508224 11233 raft_consensus.cc:2229] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:16:44.509023 11233 raft_consensus.cc:2258] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:16:44.509965 11232 tablet_replica.cc:330] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: stopping tablet replica
I20260926 02:16:44.509968 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 10789
I20260926 02:16:44.510560 11232 raft_consensus.cc:2229] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:16:44.513239 11233 ts_tablet_manager.cc:1905] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:16:44.511510 11232 raft_consensus.cc:2258] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:16:44.517928 11234 tablet_replica.cc:330] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: stopping tablet replica
I20260926 02:16:44.518416 11234 raft_consensus.cc:2229] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:16:44.518952 11234 raft_consensus.cc:2258] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:16:44.519423 11232 ts_tablet_manager.cc:1905] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:16:44.523723 11233 ts_tablet_manager.cc:1918] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:16:44.523940 11233 log.cc:1196] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/e919bebcf3844b5ca16a30128a2552e4
I20260926 02:16:44.524511 11233 ts_tablet_manager.cc:1939] T e919bebcf3844b5ca16a30128a2552e4 P a8fe6fe7f98c4205894dca7f9d8b1107: Deleting consensus metadata
I20260926 02:16:44.526897 10711 catalog_manager.cc:4841] TS a8fe6fe7f98c4205894dca7f9d8b1107 (127.28.152.131:35381): tablet e919bebcf3844b5ca16a30128a2552e4 (table test-table [id=fc456df28c634230acf6dabeb00af3a5]) successfully deleted
I20260926 02:16:44.527357 11234 ts_tablet_manager.cc:1905] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:16:44.527439 11232 ts_tablet_manager.cc:1918] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:16:44.527781 11232 log.cc:1196] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/e919bebcf3844b5ca16a30128a2552e4
I20260926 02:16:44.528398 11232 ts_tablet_manager.cc:1939] T e919bebcf3844b5ca16a30128a2552e4 P 83e3270277b9431bb9e7afa69fa0b69c: Deleting consensus metadata
I20260926 02:16:44.530472 10713 catalog_manager.cc:4841] TS 83e3270277b9431bb9e7afa69fa0b69c (127.28.152.130:45943): tablet e919bebcf3844b5ca16a30128a2552e4 (table test-table [id=fc456df28c634230acf6dabeb00af3a5]) successfully deleted
Sep 26 02:16:44 dist-test-slave-hllj krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:16:44.535036 11234 ts_tablet_manager.cc:1918] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:16:44.535244 11234 log.cc:1196] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/e919bebcf3844b5ca16a30128a2552e4
I20260926 02:16:44.535813 11234 ts_tablet_manager.cc:1939] T e919bebcf3844b5ca16a30128a2552e4 P ceb55118b8574e85950eb94585993b12: Deleting consensus metadata
I20260926 02:16:44.537876 10714 catalog_manager.cc:4841] TS ceb55118b8574e85950eb94585993b12 (127.28.152.129:46827): tablet e919bebcf3844b5ca16a30128a2552e4 (table test-table [id=fc456df28c634230acf6dabeb00af3a5]) successfully deleted
I20260926 02:16:44.538491 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.511207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.28.152.129:46827)
0926 02:16:44.511399 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:44.511405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:44.518270 (+ 6865us) server_negotiation.cc:407] Connection header received
0926 02:16:44.518594 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:44.518601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:44.518705 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:44.518781 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:44.520843 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.522061 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.524080 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.524773 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.524879 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.532053 (+ 7174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:44.532066 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:44.532082 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:44.532124 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:44.535288 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:44.536109 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:44.536118 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:44.536121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:44.536275 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:44.536690 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:44.536694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:44.536696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:44.537084 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:44.537452 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:44.537913 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 02:16:44.538136 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:16:44.539171 10914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:44.629558 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 10789
I20260926 02:16:44.649993 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 10925
Sep 26 02:16:44 dist-test-slave-hllj krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:16:44.667168 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.650839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.28.152.130:45943)
0926 02:16:44.651041 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:44.651047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:44.651068 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:16:44.651241 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:44.651247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:44.651333 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:44.651454 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:44.652773 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.653948 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.655312 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.655947 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.656081 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.660920 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:44.660938 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:44.660955 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:44.660996 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:44.663876 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:44.664836 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:44.664844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:44.664846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:44.664972 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:44.665476 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:44.665481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:44.665483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:44.665911 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:44.666226 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:44.666578 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 02:16:44.666790 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:16:44.667887 11050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:44.758738 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 10925
I20260926 02:16:44.779868 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 11060
Sep 26 02:16:44 dist-test-slave-hllj krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:16:44.799301 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:44.780737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.28.152.131:35381)
0926 02:16:44.780926 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:44.780935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:44.780992 (+ 57us) server_negotiation.cc:407] Connection header received
0926 02:16:44.781242 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:44.781245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:44.781331 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:44.781417 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:44.783336 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.784194 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.786323 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:44.786889 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:44.787008 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:44.792613 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:44.792639 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:44.792660 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:44.792702 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:44.795696 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:44.796731 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:44.796739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:44.796741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:44.796948 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:44.797511 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:44.797521 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:44.797525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:44.797958 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:44.798223 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:44.798757 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 02:16:44.798951 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:16:44.800210 11185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:44.920938 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 11060
I20260926 02:16:44.937225 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 10695
I20260926 02:16:44.938345 10758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:45.048839 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 10695
I20260926 02:16:45.066198 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:45565
--webserver_interface=127.28.152.190
--webserver_port=38427
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:45.398456 11240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:45.398762 11240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:45.398887 11240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:45.406769 11240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:16:45.406908 11240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:45.406999 11240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:45.407081 11240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:16:45.407140 11240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:16:45.418889 11240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=38427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:45.420270 11240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:45.421731 11240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:45.443818 11246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:45.443791 11247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:45.443769 11249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:45.444783 11240 server_base.cc:1034] running on GCE node
I20260926 02:16:45.445727 11240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:45.447360 11240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:45.448588 11240 hybrid_clock.cc:648] HybridClock initialized: now 1790389005448523 us; error 52 us; skew 500 ppm
I20260926 02:16:45.449011 11240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:45 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389005, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:45.452615 11240 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:16:45.454370 11240 webserver.cc:460] Webserver started at http://127.28.152.190:38427/ using document root <none> and password file <none>
I20260926 02:16:45.455168 11240 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:45.455309 11240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:45.461848 11240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:16:45.465051 11257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:45.466774 11240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:16:45.466961 11240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "3f23e98402ae4d208b61e430bc77e262"
format_stamp: "Formatted at 2026-09-26 02:16:42 on dist-test-slave-hllj"
server_key: "1e2e37a55372f855e19641d7b59b95ac"
server_key_iv: "9dbc4d2239d06af1047bcbb0afe6f9a0"
server_key_version: "encryptionkey@0"
I20260926 02:16:45.468114 11240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:45.491654 11240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:45.496299 11240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:45.496629 11240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:45.515023 11240 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:45565
I20260926 02:16:45.515058 11308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:45565 every 8 connection(s)
I20260926 02:16:45.516872 11240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:16:45.520300 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 11240
I20260926 02:16:45.521102 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:46827
--local_ip_for_outbound_sockets=127.28.152.129
--tserver_master_addrs=127.28.152.190:45565
--webserver_port=45047
--webserver_interface=127.28.152.129
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:16:45.524780 11309 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:16:45.528399 11309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Bootstrap starting.
I20260926 02:16:45.548456 11309 log.cc:824] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:45.560673 11309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:16:45.561256 11309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Bootstrap complete.
I20260926 02:16:45.570667 11309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:45.571578 11309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f23e98402ae4d208b61e430bc77e262, State: Initialized, Role: FOLLOWER
I20260926 02:16:45.572325 11309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:45.572602 11309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:16:45.572726 11309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:16:45.572901 11309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:16:45.577021 11309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:45.577637 11309 leader_election.cc:302] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f23e98402ae4d208b61e430bc77e262; no voters:
I20260926 02:16:45.578300 11309 leader_election.cc:288] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:16:45.578440 11313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:16:45.579625 11313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 2 LEADER]: Becoming Leader. State: Replica: 3f23e98402ae4d208b61e430bc77e262, State: Running, Role: LEADER
I20260926 02:16:45.580233 11313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:45.580632 11309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:16:45.582798 11315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f23e98402ae4d208b61e430bc77e262. Latest consensus state: current_term: 2 leader_uuid: "3f23e98402ae4d208b61e430bc77e262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } } }
I20260926 02:16:45.583210 11315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:45.583776 11314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3f23e98402ae4d208b61e430bc77e262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } } }
I20260926 02:16:45.584072 11314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:45.596385 11319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:16:45.605329 11319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:16:45.607263 11319 catalog_manager.cc:1259] Loaded cluster ID: 1d277b4275a34c1fa2e3d3cc0fcdcd7b
I20260926 02:16:45.607506 11319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:16:45.611047 11319 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 02:16:45.621001 11330 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:16:45.630985 11319 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 02:16:45.640645 11319 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:16:45.642393 11319 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Loaded TSK: 0
I20260926 02:16:45.643980 11319 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:16:45.871106 11311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:45.871438 11311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:45.871567 11311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:45.879273 11311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:45.879408 11311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:45.879552 11311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:16:45.889711 11311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:46827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=45047
--tserver_master_addrs=127.28.152.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:45.891031 11311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:45.892390 11311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:45.908131 11337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:45.908555 11336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:45.909547 11339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:45.910506 11311 server_base.cc:1034] running on GCE node
I20260926 02:16:45.911443 11311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:45.912875 11311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:45.914170 11311 hybrid_clock.cc:648] HybridClock initialized: now 1790389005914068 us; error 92 us; skew 500 ppm
I20260926 02:16:45.914626 11311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:45 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389005, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:45.919273 11311 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:16:45.921458 11311 webserver.cc:460] Webserver started at http://127.28.152.129:45047/ using document root <none> and password file <none>
I20260926 02:16:45.922538 11311 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:45.922721 11311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:45.931017 11311 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:16:45.935062 11346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:45.936952 11311 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:16:45.937193 11311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "ceb55118b8574e85950eb94585993b12"
format_stamp: "Formatted at 2026-09-26 02:16:43 on dist-test-slave-hllj"
server_key: "b24678412101106165e71829be238e59"
server_key_iv: "f6ad04d2f3a499c2fda5ed80a0177fee"
server_key_version: "encryptionkey@0"
I20260926 02:16:45.938694 11311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:45.958482 11311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:45.964597 11311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:45.964920 11311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:45.966248 11311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:45.967983 11311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:45.968087 11311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:45.968183 11311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:45.968276 11311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:45.999265 11311 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:46827
I20260926 02:16:45.999316 11458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:46827 every 8 connection(s)
I20260926 02:16:46.000792 11311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:16:46.005496 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 11311
I20260926 02:16:46.006108 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:45943
--local_ip_for_outbound_sockets=127.28.152.130
--tserver_master_addrs=127.28.152.190:45565
--webserver_port=46309
--webserver_interface=127.28.152.130
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:16:46 dist-test-slave-hllj krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389005, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:46.027848 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.003817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:34999)
0926 02:16:46.004596 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:46.004642 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:46.006518 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:46.008323 (+ 1805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:46.008332 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:46.008907 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:46.009610 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:46.009632 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.011911 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.011916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:46.013487 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:46.013504 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.014555 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.014561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:46.014860 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:46.015627 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:46.015656 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:46.018326 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:46.022907 (+ 4581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:46.022917 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:46.022934 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:46.023494 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:46.024154 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:46.024158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:46.024160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:46.024405 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:46.025629 (+ 1224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:46.025636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:46.026044 (+ 408us) client_negotiation.cc:769] Sending connection context
0926 02:16:46.026305 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 02:16:46.026789 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":311,"threads_started":1}
I20260926 02:16:46.028402 11461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.003610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:34999 (local address 127.28.152.190:45565)
0926 02:16:46.004193 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:46.004206 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:46.004756 (+ 550us) server_negotiation.cc:407] Connection header received
0926 02:16:46.007177 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:46.007199 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:46.007704 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:46.008129 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:46.009874 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.011673 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.013639 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.014320 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.014461 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:46.018672 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:46.018684 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:46.018706 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:46.018763 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:46.022678 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:46.023686 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:46.023697 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:46.023708 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:46.023993 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:46.024590 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:46.024595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:46.024599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:46.025424 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:46.025662 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:46.026616 (+ 954us) server_negotiation.cc:299] Negotiation successful
0926 02:16:46.027122 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260926 02:16:46.031759 11459 heartbeater.cc:347] Connected to a master server at 127.28.152.190:45565
I20260926 02:16:46.032115 11459 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:46.032899 11459 heartbeater.cc:511] Master 127.28.152.190:45565 requested a full tablet report, sending...
I20260926 02:16:46.035483 11274 ts_manager.cc:194] Registered new tserver with Master: ceb55118b8574e85950eb94585993b12 (127.28.152.129:46827)
I20260926 02:16:46.037707 11274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:34999
W20260926 02:16:46.339380 11464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:46.339689 11464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:46.339792 11464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:46.347337 11464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:46.347472 11464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:46.347615 11464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:16:46.357752 11464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:45943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=46309
--tserver_master_addrs=127.28.152.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:46.359025 11464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:46.360354 11464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:46.373682 11469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:46.373682 11470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:46.373908 11472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:46.374778 11464 server_base.cc:1034] running on GCE node
I20260926 02:16:46.375412 11464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:46.376420 11464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:46.377632 11464 hybrid_clock.cc:648] HybridClock initialized: now 1790389006377568 us; error 52 us; skew 500 ppm
I20260926 02:16:46.377974 11464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:46 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:46.381243 11464 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:16:46.382879 11464 webserver.cc:460] Webserver started at http://127.28.152.130:46309/ using document root <none> and password file <none>
I20260926 02:16:46.383591 11464 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:46.383725 11464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:46.390596 11464 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:16:46.393595 11479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:46.395121 11464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:16:46.395292 11464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "83e3270277b9431bb9e7afa69fa0b69c"
format_stamp: "Formatted at 2026-09-26 02:16:43 on dist-test-slave-hllj"
server_key: "6e3ea97c84889375759ac787736273d4"
server_key_iv: "b918b8e98ca2c60db65ca8c6e37f0378"
server_key_version: "encryptionkey@0"
I20260926 02:16:46.396402 11464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:46.411875 11464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:46.416477 11464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:46.416754 11464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:46.417858 11464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:46.419262 11464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:46.419368 11464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:46.419483 11464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:46.419552 11464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:46.449451 11464 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:45943
I20260926 02:16:46.449498 11595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:45943 every 8 connection(s)
I20260926 02:16:46.450940 11464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:16:46.451818 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 11464
I20260926 02:16:46.452600 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:35381
--local_ip_for_outbound_sockets=127.28.152.131
--tserver_master_addrs=127.28.152.190:45565
--webserver_port=36979
--webserver_interface=127.28.152.131
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:16:46 dist-test-slave-hllj krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:46.475749 11461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.454357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:43135 (local address 127.28.152.190:45565)
0926 02:16:46.454587 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:46.454594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:46.456137 (+ 1543us) server_negotiation.cc:407] Connection header received
0926 02:16:46.458906 (+ 2769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:46.458912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:46.459016 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:46.459118 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:46.461506 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.462495 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.464336 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.464938 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.465096 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:46.469036 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:46.469052 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:46.469058 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:46.469104 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:46.472227 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:46.473078 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:46.473084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:46.473088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:46.473288 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:46.473792 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:46.473797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:46.473801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:46.474192 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:46.474353 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:46.475149 (+ 796us) server_negotiation.cc:299] Negotiation successful
0926 02:16:46.475407 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:16:46.476567 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.454972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:43135)
0926 02:16:46.455915 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:46.455984 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:46.458442 (+ 2458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:46.459441 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:46.459454 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:46.460272 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:46.461253 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:46.461280 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.462698 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.462703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:46.464136 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:46.464153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.465248 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.465256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:46.465482 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:46.466237 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:46.466262 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:46.468756 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:46.472406 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:46.472413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:46.472429 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:46.472936 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:46.473450 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:46.473455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:46.473457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:46.473665 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:46.474337 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:46.474344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:46.474739 (+ 395us) client_negotiation.cc:769] Sending connection context
0926 02:16:46.474982 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 02:16:46.475411 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":331,"threads_started":1}
I20260926 02:16:46.478077 11596 heartbeater.cc:347] Connected to a master server at 127.28.152.190:45565
I20260926 02:16:46.478469 11596 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:46.479408 11596 heartbeater.cc:511] Master 127.28.152.190:45565 requested a full tablet report, sending...
I20260926 02:16:46.481297 11274 ts_manager.cc:194] Registered new tserver with Master: 83e3270277b9431bb9e7afa69fa0b69c (127.28.152.130:45943)
I20260926 02:16:46.482188 11274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:43135
W20260926 02:16:46.789270 11599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:46.789611 11599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:46.789749 11599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:46.797313 11599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:46.797444 11599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:46.797613 11599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:16:46.808202 11599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:35381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=36979
--tserver_master_addrs=127.28.152.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:46.809613 11599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:46.811033 11599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:46.824878 11605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:46.825040 11604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:46.824942 11607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:46.826200 11599 server_base.cc:1034] running on GCE node
I20260926 02:16:46.826912 11599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:46.828027 11599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:46.829252 11599 hybrid_clock.cc:648] HybridClock initialized: now 1790389006829187 us; error 71 us; skew 500 ppm
I20260926 02:16:46.829629 11599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:46 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:46.833110 11599 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:16:46.834810 11599 webserver.cc:460] Webserver started at http://127.28.152.131:36979/ using document root <none> and password file <none>
I20260926 02:16:46.835520 11599 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:46.835657 11599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:46.841598 11599 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 02:16:46.844626 11614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:46.846182 11599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:16:46.846350 11599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "a8fe6fe7f98c4205894dca7f9d8b1107"
format_stamp: "Formatted at 2026-09-26 02:16:44 on dist-test-slave-hllj"
server_key: "9bd76286cf567925de5e176ac25ee91a"
server_key_iv: "edca37cddb90db8d1228561db370ca58"
server_key_version: "encryptionkey@0"
I20260926 02:16:46.847350 11599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:46.875885 11599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:46.880030 11599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:46.880296 11599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:46.881474 11599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:46.882934 11599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:46.883033 11599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:46.883142 11599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:46.883212 11599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:46.914906 11599 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:35381
I20260926 02:16:46.914966 11728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:35381 every 8 connection(s)
I20260926 02:16:46.916404 11599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:16:46.919275 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 11599
Sep 26 02:16:46 dist-test-slave-hllj krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:16:46.940272 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.920215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.28.152.190:45565)
0926 02:16:46.924428 (+ 4213us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:46.924432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:46.924447 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:16:46.924499 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:46.924503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:46.924555 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:46.924637 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:46.926702 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.927940 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.930003 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.930611 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.930743 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:46.934173 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:46.934181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:46.934186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:46.934235 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:46.937360 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:46.938155 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:46.938160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:46.938163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:46.938319 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:46.938747 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:46.938751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:46.938753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:46.939048 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:46.939314 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:46.939661 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 02:16:46.939897 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":4113,"thread_start_us":4014,"threads_started":1}
I20260926 02:16:46.941641 11461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.919199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:36965 (local address 127.28.152.190:45565)
0926 02:16:46.919431 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:46.919436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:46.920880 (+ 1444us) server_negotiation.cc:407] Connection header received
0926 02:16:46.924037 (+ 3157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:46.924041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:46.924128 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:46.924239 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:46.926651 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.927937 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.930324 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.931028 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.931118 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:46.935299 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:46.935304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:46.935307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:46.935356 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:46.938260 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:46.939096 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:46.939100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:46.939102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:46.939238 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:46.939688 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:46.939692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:46.939694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:46.939957 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:46.940278 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:46.941030 (+ 752us) server_negotiation.cc:299] Negotiation successful
0926 02:16:46.941284 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":76}
I20260926 02:16:46.942761 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.919749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:36965)
0926 02:16:46.920676 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:46.920734 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:46.923614 (+ 2880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:46.924597 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:46.924610 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:46.925456 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:16:46.926383 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:46.926414 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.928154 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.928160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:46.930181 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:46.930195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.931278 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.931286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:46.931559 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:46.932174 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:16:46.932205 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:16:46.935026 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:16:46.938443 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:46.938451 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:46.938467 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:46.938966 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:46.939348 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:16:46.939352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:16:46.939354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:16:46.939567 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:16:46.940097 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:16:46.940106 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:16:46.940606 (+ 500us) client_negotiation.cc:769] Sending connection context
0926 02:16:46.940923 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 02:16:46.941539 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":340,"threads_started":1}
I20260926 02:16:46.944108 11729 heartbeater.cc:347] Connected to a master server at 127.28.152.190:45565
I20260926 02:16:46.944404 11729 heartbeater.cc:464] Registering TS with master...
I20260926 02:16:46.945227 11729 heartbeater.cc:511] Master 127.28.152.190:45565 requested a full tablet report, sending...
I20260926 02:16:46.946640 11274 ts_manager.cc:194] Registered new tserver with Master: a8fe6fe7f98c4205894dca7f9d8b1107 (127.28.152.131:35381)
I20260926 02:16:46.947449 11274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:36965
I20260926 02:16:46.955061 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:16:46.962082 11461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.956015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.28.152.190:45565)
0926 02:16:46.956239 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:46.956246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:46.956580 (+ 334us) server_negotiation.cc:407] Connection header received
0926 02:16:46.956820 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:46.956824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:46.956979 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:46.957115 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:16:46.957523 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.958273 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.959826 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:46.960258 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:46.960354 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:46.960843 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:16:46.961454 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:16:46.961599 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:46.961715 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 02:16:46.961804 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:16:46.963145 11274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:16:46.970609 11274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:16:47.013721 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.983816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.28.152.130:45943)
0926 02:16:46.993698 (+ 9882us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:46.993710 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:47.003664 (+ 9954us) server_negotiation.cc:407] Connection header received
0926 02:16:47.004944 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:47.004953 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:47.005264 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:47.005471 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:47.006352 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.007810 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.010771 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.012268 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.012427 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.012617 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:47.013232 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 02:16:47.013421 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9578,"thread_start_us":9465,"threads_started":1}
I20260926 02:16:47.013823 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.983232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:45943 (local address 127.0.0.1:35446)
0926 02:16:47.003468 (+ 20236us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:47.003496 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:47.004695 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:47.005733 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:47.005738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:47.005763 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:47.006133 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:47.006146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.007988 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.007992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:47.010601 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:47.010618 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.012653 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.012662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:47.012924 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.012991 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 02:16:47.013161 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 02:16:47.013395 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20055,"mutex_wait_us":152,"thread_start_us":25700,"threads_started":1}
I20260926 02:16:47.015332 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.983120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46827 (local address 127.0.0.1:48160)
0926 02:16:46.994544 (+ 11424us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:46.994596 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:46.994833 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:47.004951 (+ 10118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:47.004960 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:47.005002 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:47.005453 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:47.005468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.007782 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.007787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:47.010603 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:47.010618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.014488 (+ 3870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.014494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:47.014700 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.014732 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:16:47.014803 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 02:16:47.014940 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11089,"mutex_wait_us":53,"thread_start_us":20995,"threads_started":1}
I20260926 02:16:47.017060 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.983705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:35381 (local address 127.0.0.1:54898)
0926 02:16:47.009243 (+ 25538us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:47.009263 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:47.009415 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:47.009949 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:47.009953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:47.009976 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:47.010380 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:47.010391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.012315 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.012321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:47.014741 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:47.014757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.016269 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.016275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:47.016449 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.016481 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:16:47.016572 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 02:16:47.016700 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25355,"mutex_wait_us":86,"thread_start_us":9441,"threads_started":1}
I20260926 02:16:47.017117 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.984355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.28.152.131:35381)
0926 02:16:47.008515 (+ 24160us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:47.008524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:47.009379 (+ 855us) server_negotiation.cc:407] Connection header received
0926 02:16:47.009556 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:47.009564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:47.009760 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:47.009928 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:47.010565 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.012175 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.014926 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.016036 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.016220 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.016386 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:47.016630 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 02:16:47.016801 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23972,"thread_start_us":24170,"threads_started":1}
I20260926 02:16:47.018029 11528 tablet_service.cc:1467] Processing CreateTablet for tablet acb6224c87604e7dbc9ebc56fda4aa65 (DEFAULT_TABLE table=test-table [id=4d633c71d74c4746885a8351c68484cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:16:47.019311 11528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb6224c87604e7dbc9ebc56fda4aa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:47.021706 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:46.983211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.28.152.129:46827)
0926 02:16:47.004225 (+ 21014us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:47.004233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:47.004263 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:16:47.004366 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:47.004374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:47.004674 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:47.005675 (+ 1001us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:47.005957 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.007614 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.012720 (+ 5106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.014250 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.014447 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.014628 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:47.014894 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 02:16:47.021369 (+ 6475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20839,"thread_start_us":20673,"threads_started":1}
I20260926 02:16:47.024523 11662 tablet_service.cc:1467] Processing CreateTablet for tablet acb6224c87604e7dbc9ebc56fda4aa65 (DEFAULT_TABLE table=test-table [id=4d633c71d74c4746885a8351c68484cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:16:47.025735 11662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb6224c87604e7dbc9ebc56fda4aa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:47.025506 11394 tablet_service.cc:1467] Processing CreateTablet for tablet acb6224c87604e7dbc9ebc56fda4aa65 (DEFAULT_TABLE table=test-table [id=4d633c71d74c4746885a8351c68484cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:16:47.026886 11394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb6224c87604e7dbc9ebc56fda4aa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:16:47.036494 11746 tablet_bootstrap.cc:492] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: Bootstrap starting.
I20260926 02:16:47.040565 11746 tablet_bootstrap.cc:654] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:47.041206 11459 heartbeater.cc:503] Master 127.28.152.190:45565 was elected leader, sending a full tablet report...
I20260926 02:16:47.043181 11746 log.cc:824] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:47.045960 11747 tablet_bootstrap.cc:492] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: Bootstrap starting.
I20260926 02:16:47.047149 11748 tablet_bootstrap.cc:492] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: Bootstrap starting.
I20260926 02:16:47.049685 11747 tablet_bootstrap.cc:654] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:47.050734 11746 tablet_bootstrap.cc:492] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: No bootstrap required, opened a new log
I20260926 02:16:47.050787 11748 tablet_bootstrap.cc:654] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: Neither blocks nor log segments found. Creating new log.
I20260926 02:16:47.050771 11747 log.cc:824] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:47.051041 11746 ts_tablet_manager.cc:1397] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260926 02:16:47.052151 11748 log.cc:824] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:47.054031 11747 tablet_bootstrap.cc:492] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: No bootstrap required, opened a new log
I20260926 02:16:47.054293 11747 ts_tablet_manager.cc:1397] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 02:16:47.055781 11748 tablet_bootstrap.cc:492] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: No bootstrap required, opened a new log
I20260926 02:16:47.056061 11748 ts_tablet_manager.cc:1397] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 02:16:47.061596 11746 raft_consensus.cc:348] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } }
I20260926 02:16:47.062215 11746 raft_consensus.cc:374] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:47.062322 11746 raft_consensus.cc:729] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e3270277b9431bb9e7afa69fa0b69c, State: Initialized, Role: FOLLOWER
I20260926 02:16:47.062036 11747 raft_consensus.cc:348] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } }
I20260926 02:16:47.062492 11747 raft_consensus.cc:374] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:47.062587 11747 raft_consensus.cc:729] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fe6fe7f98c4205894dca7f9d8b1107, State: Initialized, Role: FOLLOWER
I20260926 02:16:47.063047 11746 consensus_queue.cc:260] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } }
I20260926 02:16:47.063161 11747 consensus_queue.cc:260] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } }
I20260926 02:16:47.064311 11729 heartbeater.cc:503] Master 127.28.152.190:45565 was elected leader, sending a full tablet report...
I20260926 02:16:47.064827 11596 heartbeater.cc:503] Master 127.28.152.190:45565 was elected leader, sending a full tablet report...
I20260926 02:16:47.065207 11747 ts_tablet_manager.cc:1428] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260926 02:16:47.066764 11746 ts_tablet_manager.cc:1428] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260926 02:16:47.067126 11748 raft_consensus.cc:348] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } }
I20260926 02:16:47.067613 11748 raft_consensus.cc:374] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:16:47.067719 11748 raft_consensus.cc:729] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb55118b8574e85950eb94585993b12, State: Initialized, Role: FOLLOWER
I20260926 02:16:47.068480 11748 consensus_queue.cc:260] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } }
I20260926 02:16:47.071219 11748 ts_tablet_manager.cc:1428] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260926 02:16:47.168048 11730 tablet.cc:2369] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:16:47.203380 11597 tablet.cc:2369] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:16:47.204898 11752 raft_consensus.cc:482] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:16:47.205214 11752 raft_consensus.cc:504] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } }
I20260926 02:16:47.206846 11752 leader_election.cc:288] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ceb55118b8574e85950eb94585993b12 (127.28.152.129:46827), 83e3270277b9431bb9e7afa69fa0b69c (127.28.152.130:45943)
I20260926 02:16:47.212512 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.207380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:59919 (local address 127.28.152.129:46827)
0926 02:16:47.207614 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:47.207620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:47.207706 (+ 86us) server_negotiation.cc:407] Connection header received
0926 02:16:47.207903 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:47.207908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:47.207969 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:47.208043 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:47.208502 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.209270 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.210883 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.211662 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.211812 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.211888 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:47.212140 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 02:16:47.212267 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 02:16:47.212541 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.207162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46827 (local address 127.28.152.131:59919)
0926 02:16:47.207603 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:47.207624 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:47.207769 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:47.208113 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:47.208118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:47.208147 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:47.208365 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:47.208374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.209420 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.209423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:47.210731 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:47.210743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.211812 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.211818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:47.211990 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.212013 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:16:47.212100 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 02:16:47.212240 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"spinlock_wait_cycles":40832}
I20260926 02:16:47.213053 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.207356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:33283 (local address 127.28.152.130:45943)
0926 02:16:47.207562 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:47.207621 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:47.208199 (+ 578us) server_negotiation.cc:407] Connection header received
0926 02:16:47.208400 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:47.208404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:47.208462 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:47.208534 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:16:47.209057 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.209764 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.211393 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.212176 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.212315 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.212388 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:47.212712 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 02:16:47.212844 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:16:47.213616 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.207372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:45943 (local address 127.28.152.131:33283)
0926 02:16:47.208091 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:47.208106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:16:47.208267 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:16:47.208573 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:16:47.208578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:16:47.208600 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:16:47.208902 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:47.208910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.209917 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.209920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:47.211231 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:16:47.211243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.212378 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.212384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:16:47.212528 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.212550 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 02:16:47.213257 (+ 707us) client_negotiation.cc:240] Negotiation successful
0926 02:16:47.213373 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":398,"threads_started":1}
I20260926 02:16:47.213584 11414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb6224c87604e7dbc9ebc56fda4aa65" candidate_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb55118b8574e85950eb94585993b12" is_pre_election: true
I20260926 02:16:47.214121 11414 raft_consensus.cc:2454] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fe6fe7f98c4205894dca7f9d8b1107 in term 0.
I20260926 02:16:47.214571 11548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb6224c87604e7dbc9ebc56fda4aa65" candidate_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e3270277b9431bb9e7afa69fa0b69c" is_pre_election: true
I20260926 02:16:47.214769 11618 leader_election.cc:302] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8fe6fe7f98c4205894dca7f9d8b1107, ceb55118b8574e85950eb94585993b12; no voters:
I20260926 02:16:47.215024 11548 raft_consensus.cc:2454] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8fe6fe7f98c4205894dca7f9d8b1107 in term 0.
I20260926 02:16:47.215148 11752 raft_consensus.cc:2781] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:16:47.215288 11752 raft_consensus.cc:482] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:16:47.215386 11752 raft_consensus.cc:3037] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:47.217702 11752 raft_consensus.cc:504] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } }
I20260926 02:16:47.218318 11752 leader_election.cc:288] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [CANDIDATE]: Term 1 election: Requested vote from peers ceb55118b8574e85950eb94585993b12 (127.28.152.129:46827), 83e3270277b9431bb9e7afa69fa0b69c (127.28.152.130:45943)
I20260926 02:16:47.218796 11414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb6224c87604e7dbc9ebc56fda4aa65" candidate_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb55118b8574e85950eb94585993b12"
I20260926 02:16:47.218825 11548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb6224c87604e7dbc9ebc56fda4aa65" candidate_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e3270277b9431bb9e7afa69fa0b69c"
I20260926 02:16:47.218986 11414 raft_consensus.cc:3037] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:47.218999 11548 raft_consensus.cc:3037] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:16:47.221223 11414 raft_consensus.cc:2454] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fe6fe7f98c4205894dca7f9d8b1107 in term 1.
I20260926 02:16:47.221306 11548 raft_consensus.cc:2454] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8fe6fe7f98c4205894dca7f9d8b1107 in term 1.
I20260926 02:16:47.221632 11618 leader_election.cc:302] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8fe6fe7f98c4205894dca7f9d8b1107, ceb55118b8574e85950eb94585993b12; no voters:
I20260926 02:16:47.221909 11752 raft_consensus.cc:2781] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:16:47.222332 11752 raft_consensus.cc:686] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 1 LEADER]: Becoming Leader. State: Replica: a8fe6fe7f98c4205894dca7f9d8b1107, State: Running, Role: LEADER
I20260926 02:16:47.222777 11752 consensus_queue.cc:237] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } } peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } }
I20260926 02:16:47.227195 11274 catalog_manager.cc:5495] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8fe6fe7f98c4205894dca7f9d8b1107 (127.28.152.131). New cstate: current_term: 1 leader_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8fe6fe7f98c4205894dca7f9d8b1107" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 35381 } health_report { overall_health: HEALTHY } } }
W20260926 02:16:47.253046 11460 tablet.cc:2369] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:16:47.255798 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.250224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.28.152.131:35381)
0926 02:16:47.250389 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:47.250393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:47.250455 (+ 62us) server_negotiation.cc:407] Connection header received
0926 02:16:47.250676 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:47.250680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:47.250766 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:47.250888 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:16:47.251174 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.251956 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.253827 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.254224 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.254318 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.254805 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:16:47.255303 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:16:47.255410 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:47.255470 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 02:16:47.255539 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 02:16:47.264182 11414 raft_consensus.cc:1261] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12 [term 1 FOLLOWER]: Refusing update from remote peer a8fe6fe7f98c4205894dca7f9d8b1107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:16:47.264181 11548 raft_consensus.cc:1261] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c [term 1 FOLLOWER]: Refusing update from remote peer a8fe6fe7f98c4205894dca7f9d8b1107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:16:47.265195 11756 consensus_queue.cc:1035] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb55118b8574e85950eb94585993b12" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:16:47.265633 11752 consensus_queue.cc:1035] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e3270277b9431bb9e7afa69fa0b69c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:16:47.283020 11759 mvcc.cc:205] Tried to move back new op lower bound from 7333433373743149056 to 7333433373588013056. Current Snapshot: MvccSnapshot[applied={T|T < 7333433373743149056}]
I20260926 02:16:47.292363 11274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:16:47.292635 11274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:16:47.296144 11274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Sending DeleteTablet for 3 replicas of tablet acb6224c87604e7dbc9ebc56fda4aa65
I20260926 02:16:47.297493 11394 tablet_service.cc:1514] Processing DeleteTablet for tablet acb6224c87604e7dbc9ebc56fda4aa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:16:47 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:48160
I20260926 02:16:47.297559 11528 tablet_service.cc:1514] Processing DeleteTablet for tablet acb6224c87604e7dbc9ebc56fda4aa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:16:47 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:35446
I20260926 02:16:47.297550 11662 tablet_service.cc:1514] Processing DeleteTablet for tablet acb6224c87604e7dbc9ebc56fda4aa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:16:47 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:54898
I20260926 02:16:47.300902 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 11311
I20260926 02:16:47.311384 11769 tablet_replica.cc:330] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: stopping tablet replica
I20260926 02:16:47.312029 11767 tablet_replica.cc:330] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: stopping tablet replica
I20260926 02:16:47.312664 11767 raft_consensus.cc:2229] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:16:47.313256 11769 raft_consensus.cc:2229] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:16:47.313879 11767 raft_consensus.cc:2258] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:16:47.314967 11768 tablet_replica.cc:330] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: stopping tablet replica
I20260926 02:16:47.314956 11769 raft_consensus.cc:2258] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:16:47.315775 11768 raft_consensus.cc:2229] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:16:47.316730 11768 raft_consensus.cc:2258] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:16:47.317921 11767 ts_tablet_manager.cc:1905] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:16:47.318439 11769 ts_tablet_manager.cc:1905] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:16:47.322119 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.302629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.28.152.129:46827)
0926 02:16:47.302840 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:47.302847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:47.302912 (+ 65us) server_negotiation.cc:407] Connection header received
0926 02:16:47.303109 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:47.303114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:47.303209 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:47.303328 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:47.305230 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.306434 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.308670 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.309416 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.309561 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.314358 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:47.314374 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:47.314405 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:47.314454 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:47.318828 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:47.319644 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:47.319658 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:47.319662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:47.319866 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:47.320305 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:47.320314 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:47.320317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:47.320797 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:47.321160 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:47.321449 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 02:16:47.321708 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:16:47.323091 11454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:47.325059 11768 ts_tablet_manager.cc:1905] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:16:47.328693 11769 ts_tablet_manager.cc:1918] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:16:47.328918 11769 log.cc:1196] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/acb6224c87604e7dbc9ebc56fda4aa65
I20260926 02:16:47.329998 11769 ts_tablet_manager.cc:1939] T acb6224c87604e7dbc9ebc56fda4aa65 P 83e3270277b9431bb9e7afa69fa0b69c: Deleting consensus metadata
I20260926 02:16:47.332391 11260 catalog_manager.cc:4841] TS 83e3270277b9431bb9e7afa69fa0b69c (127.28.152.130:45943): tablet acb6224c87604e7dbc9ebc56fda4aa65 (table test-table [id=4d633c71d74c4746885a8351c68484cb]) successfully deleted
I20260926 02:16:47.332821 11768 ts_tablet_manager.cc:1918] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:16:47.333177 11768 log.cc:1196] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/acb6224c87604e7dbc9ebc56fda4aa65
I20260926 02:16:47.334072 11768 ts_tablet_manager.cc:1939] T acb6224c87604e7dbc9ebc56fda4aa65 P a8fe6fe7f98c4205894dca7f9d8b1107: Deleting consensus metadata
I20260926 02:16:47.336709 11258 catalog_manager.cc:4841] TS a8fe6fe7f98c4205894dca7f9d8b1107 (127.28.152.131:35381): tablet acb6224c87604e7dbc9ebc56fda4aa65 (table test-table [id=4d633c71d74c4746885a8351c68484cb]) successfully deleted
I20260926 02:16:47.426247 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 11311
W20260926 02:16:47.447496 11261 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.152.129:46827 (error 108)
I20260926 02:16:47.447886 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 11464
I20260926 02:16:47.450990 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.450304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46827 (local address 127.0.0.1:48176)
0926 02:16:47.450698 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:47.450821 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271}
W20260926 02:16:47.451282 11261 catalog_manager.cc:4584] TS ceb55118b8574e85950eb94585993b12 (127.28.152.129:46827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet acb6224c87604e7dbc9ebc56fda4aa65: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46827: connect: Connection refused (error 111)
I20260926 02:16:47.471711 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.448778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.28.152.130:45943)
0926 02:16:47.449268 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:47.449274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:47.449296 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:16:47.449367 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:47.449372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:47.449465 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:47.449583 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:47.451635 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.452827 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.454928 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.455600 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.455726 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.464169 (+ 8443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:47.464187 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:47.464213 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:47.464262 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:47.468338 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:47.469376 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:47.469387 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:47.469390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:47.469565 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:47.469981 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:47.469986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:47.469990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:47.470523 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:47.470812 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:47.471119 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 02:16:47.471354 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365}
I20260926 02:16:47.472441 11590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:47.505191 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.504532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46827 (local address 127.0.0.1:48180)
0926 02:16:47.504806 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:47.504934 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 02:16:47.506400 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.505990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46827 (local address 127.0.0.1:48184)
0926 02:16:47.506148 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:47.506255 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 02:16:47.567209 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.566628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46827 (local address 127.0.0.1:48190)
0926 02:16:47.566900 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:47.567027 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 02:16:47.568462 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.568087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46827 (local address 127.0.0.1:48200)
0926 02:16:47.568229 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:47.568314 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 02:16:47.577396 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 11464
I20260926 02:16:47.591647 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 11599
I20260926 02:16:47.607281 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.592443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.28.152.131:35381)
0926 02:16:47.592662 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:16:47.592668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:16:47.592687 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:16:47.592831 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:16:47.592837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:16:47.592914 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:16:47.593006 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:16:47.594451 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.595303 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.596754 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:16:47.597320 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:16:47.597449 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:16:47.600367 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:16:47.600388 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:16:47.600406 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:16:47.600458 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:16:47.603853 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:47.604716 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:47.604724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:47.604726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:47.604899 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:16:47.605530 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:16:47.605534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:16:47.605537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:16:47.605959 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:16:47.606275 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 02:16:47.606642 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 02:16:47.606864 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:16:47.608120 11724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:47.649562 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.648893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46827 (local address 127.0.0.1:48202)
0926 02:16:47.649179 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:47.649297 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 02:16:47.650753 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:47.650273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:46827 (local address 127.0.0.1:48216)
0926 02:16:47.650537 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:47.650629 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 02:16:47.735379 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 11599
I20260926 02:16:47.751994 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 11240
I20260926 02:16:47.753242 11304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:16:47.865702 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 11240
I20260926 02:16:47.883476 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:45565
--webserver_interface=127.28.152.190
--webserver_port=38427
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:48.217556 11775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:48.217866 11775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:48.217984 11775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:48.225373 11775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:16:48.225494 11775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:48.225564 11775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:48.225639 11775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:16:48.225694 11775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:16:48.235973 11775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=38427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:48.237399 11775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:48.238723 11775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:48.250087 11780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:48.250135 11781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:48.250303 11783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:48.251041 11775 server_base.cc:1034] running on GCE node
I20260926 02:16:48.251888 11775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:48.253346 11775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:48.254568 11775 hybrid_clock.cc:648] HybridClock initialized: now 1790389008254510 us; error 54 us; skew 500 ppm
I20260926 02:16:48.254945 11775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:48 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389008, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:48.258001 11775 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:16:48.259469 11775 webserver.cc:460] Webserver started at http://127.28.152.190:38427/ using document root <none> and password file <none>
I20260926 02:16:48.260169 11775 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:48.260286 11775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:48.265988 11775 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 02:16:48.268852 11790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:48.270418 11775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:16:48.270592 11775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "3f23e98402ae4d208b61e430bc77e262"
format_stamp: "Formatted at 2026-09-26 02:16:42 on dist-test-slave-hllj"
server_key: "1e2e37a55372f855e19641d7b59b95ac"
server_key_iv: "9dbc4d2239d06af1047bcbb0afe6f9a0"
server_key_version: "encryptionkey@0"
I20260926 02:16:48.271589 11775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:48.308388 11775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:48.312942 11775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:48.313254 11775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:48.334877 11775 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:45565
I20260926 02:16:48.334884 11841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:45565 every 8 connection(s)
I20260926 02:16:48.336459 11775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:16:48.343849 11842 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:16:48.346438 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 11775
I20260926 02:16:48.347469 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:46827
--local_ip_for_outbound_sockets=127.28.152.129
--tserver_master_addrs=127.28.152.190:45565
--webserver_port=45047
--webserver_interface=127.28.152.129
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:16:48.348065 11842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Bootstrap starting.
I20260926 02:16:48.370648 11842 log.cc:824] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Log is configured to *not* fsync() on all Append() calls
I20260926 02:16:48.388008 11842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:16:48.388556 11842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262: Bootstrap complete.
I20260926 02:16:48.398727 11842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:48.399652 11842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f23e98402ae4d208b61e430bc77e262, State: Initialized, Role: FOLLOWER
I20260926 02:16:48.400391 11842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:48.400633 11842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:16:48.400750 11842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:16:48.400921 11842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:16:48.404978 11842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:48.405542 11842 leader_election.cc:302] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f23e98402ae4d208b61e430bc77e262; no voters:
I20260926 02:16:48.406224 11842 leader_election.cc:288] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 02:16:48.406380 11846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 02:16:48.407393 11846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [term 3 LEADER]: Becoming Leader. State: Replica: 3f23e98402ae4d208b61e430bc77e262, State: Running, Role: LEADER
I20260926 02:16:48.408083 11846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } }
I20260926 02:16:48.408546 11842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:16:48.410948 11847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3f23e98402ae4d208b61e430bc77e262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } } }
I20260926 02:16:48.411507 11847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:48.410952 11848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f23e98402ae4d208b61e430bc77e262. Latest consensus state: current_term: 3 leader_uuid: "3f23e98402ae4d208b61e430bc77e262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23e98402ae4d208b61e430bc77e262" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45565 } } }
I20260926 02:16:48.412236 11848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f23e98402ae4d208b61e430bc77e262 [sys.catalog]: This master's current role is: LEADER
I20260926 02:16:48.414985 11854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:16:48.424008 11854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:16:48.425679 11854 catalog_manager.cc:1259] Loaded cluster ID: 1d277b4275a34c1fa2e3d3cc0fcdcd7b
I20260926 02:16:48.425860 11854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 02:16:48.427935 11854 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f91ec3119ad google::LogMessage::Fail() at ??:0
@ 0x7f91ec31488a google::LogMessage::SendToLog() at ??:0
@ 0x7f91ec3114de google::LogMessage::Flush() at ??:0
@ 0x7f91ec312a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9201d44512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9201d41976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9201dc9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9201dc9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f91eef5d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f91eef6091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f91eef60799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f91eef3beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f91f22156db start_thread at ??:0
@ 0x7f91eb1ab61f clone at ??:0
W20260926 02:16:48.740842 11844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:48.741164 11844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:48.741302 11844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:48.748831 11844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:48.748951 11844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:48.749107 11844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:16:48.759186 11844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:46827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=45047
--tserver_master_addrs=127.28.152.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:48.760514 11844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:48.761888 11844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:48.775624 11886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:48.775839 11888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:48.776072 11885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:48.776563 11844 server_base.cc:1034] running on GCE node
I20260926 02:16:48.777276 11844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:48.778283 11844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:48.779529 11844 hybrid_clock.cc:648] HybridClock initialized: now 1790389008779450 us; error 58 us; skew 500 ppm
I20260926 02:16:48.779865 11844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:48 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389008, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:48.783219 11844 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:16:48.784667 11844 webserver.cc:460] Webserver started at http://127.28.152.129:45047/ using document root <none> and password file <none>
I20260926 02:16:48.785429 11844 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:48.785552 11844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:48.791163 11844 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:16:48.796698 11895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:48.798200 11844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:16:48.798370 11844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "ceb55118b8574e85950eb94585993b12"
format_stamp: "Formatted at 2026-09-26 02:16:43 on dist-test-slave-hllj"
server_key: "b24678412101106165e71829be238e59"
server_key_iv: "f6ad04d2f3a499c2fda5ed80a0177fee"
server_key_version: "encryptionkey@0"
I20260926 02:16:48.799376 11844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:48.818092 11844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:48.822467 11844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:48.822736 11844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:48.823900 11844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:48.825896 11902 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 02:16:48.831504 11902 ts_tablet_manager.cc:1852] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 02:16:48.832536 11902 ts_tablet_manager.cc:1905] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:16:48.840279 11902 ts_tablet_manager.cc:1918] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:16:48.840485 11902 log.cc:1196] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/acb6224c87604e7dbc9ebc56fda4aa65
I20260926 02:16:48.840961 11902 ts_tablet_manager.cc:1939] T acb6224c87604e7dbc9ebc56fda4aa65 P ceb55118b8574e85950eb94585993b12: Deleting consensus metadata
I20260926 02:16:48.841398 11844 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 02:16:48.841522 11844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260926 02:16:48.841652 11844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:48.841724 11844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:48.872856 11844 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:46827
I20260926 02:16:48.872898 12008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:46827 every 8 connection(s)
I20260926 02:16:48.874279 11844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:16:48.879403 12011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:48.877461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:44461)
0926 02:16:48.878203 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:48.878463 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":279,"threads_started":1}
W20260926 02:16:48.880044 12009 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:45565 (0 consecutive failures): Network error: Failed to ping master at 127.28.152.190:45565: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
I20260926 02:16:48.880873 12011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:48.880540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:53743)
0926 02:16:48.880693 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:48.880765 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 02:16:48.881743 12011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:48.881415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:37447)
0926 02:16:48.881532 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:48.881626 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 02:16:48.881949 12009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:16:48.884703 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 11844
I20260926 02:16:48.885390 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:45943
--local_ip_for_outbound_sockets=127.28.152.130
--tserver_master_addrs=127.28.152.190:45565
--webserver_port=46309
--webserver_interface=127.28.152.130
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:49.212728 12012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:49.213014 12012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:49.213109 12012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:49.220387 12012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:49.220516 12012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:49.220639 12012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:16:49.230528 12012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:45943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=46309
--tserver_master_addrs=127.28.152.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:49.231777 12012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:49.233081 12012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:49.246495 12020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:49.246505 12018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:49.247190 12017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:49.247328 12012 server_base.cc:1034] running on GCE node
I20260926 02:16:49.248003 12012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:49.248943 12012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:49.250180 12012 hybrid_clock.cc:648] HybridClock initialized: now 1790389009250140 us; error 63 us; skew 500 ppm
I20260926 02:16:49.250514 12012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:49 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389009, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:49.253844 12012 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:16:49.255345 12012 webserver.cc:460] Webserver started at http://127.28.152.130:46309/ using document root <none> and password file <none>
I20260926 02:16:49.256049 12012 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:49.256176 12012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:49.261819 12012 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:16:49.264772 12027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:49.266425 12012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:16:49.266594 12012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "83e3270277b9431bb9e7afa69fa0b69c"
format_stamp: "Formatted at 2026-09-26 02:16:43 on dist-test-slave-hllj"
server_key: "6e3ea97c84889375759ac787736273d4"
server_key_iv: "b918b8e98ca2c60db65ca8c6e37f0378"
server_key_version: "encryptionkey@0"
I20260926 02:16:49.267629 12012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:49.290607 12012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:49.295017 12012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:49.295272 12012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:49.296312 12012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:49.297734 12012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:49.297822 12012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:49.297904 12012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:49.297955 12012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:49.329097 12012 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:45943
I20260926 02:16:49.329200 12141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:45943 every 8 connection(s)
I20260926 02:16:49.330631 12012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:16:49.335604 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:49.333668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:53891)
0926 02:16:49.334400 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:49.334644 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":272,"threads_started":1}
W20260926 02:16:49.336247 12142 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:45565 (0 consecutive failures): Network error: Failed to ping master at 127.28.152.190:45565: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
I20260926 02:16:49.337093 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:49.336766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:50245)
0926 02:16:49.336908 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:49.336980 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 02:16:49.337647 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 12012
I20260926 02:16:49.338054 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:49.337672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:41985)
0926 02:16:49.337826 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:49.337917 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 02:16:49.338295 12142 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:16:49.338376 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:35381
--local_ip_for_outbound_sockets=127.28.152.131
--tserver_master_addrs=127.28.152.190:45565
--webserver_port=36979
--webserver_interface=127.28.152.131
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:16:49.664592 12145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:16:49.664891 12145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:16:49.664989 12145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:16:49.672569 12145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:16:49.672706 12145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:16:49.672844 12145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:16:49.682904 12145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:35381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=36979
--tserver_master_addrs=127.28.152.190:45565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:16:49.684214 12145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:16:49.685599 12145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:16:49.699016 12150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:49.699106 12151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:16:49.699074 12153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:16:49.699730 12145 server_base.cc:1034] running on GCE node
I20260926 02:16:49.700367 12145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:16:49.701346 12145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:16:49.702554 12145 hybrid_clock.cc:648] HybridClock initialized: now 1790389009702501 us; error 45 us; skew 500 ppm
I20260926 02:16:49.702854 12145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:16:49 dist-test-slave-hllj krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389009, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:16:49.706033 12145 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:16:49.707561 12145 webserver.cc:460] Webserver started at http://127.28.152.131:36979/ using document root <none> and password file <none>
I20260926 02:16:49.708243 12145 fs_manager.cc:359] Metadata directory not provided
I20260926 02:16:49.708345 12145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:16:49.713991 12145 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 02:16:49.716837 12160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:49.718669 12145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:16:49.718865 12145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "a8fe6fe7f98c4205894dca7f9d8b1107"
format_stamp: "Formatted at 2026-09-26 02:16:44 on dist-test-slave-hllj"
server_key: "9bd76286cf567925de5e176ac25ee91a"
server_key_iv: "edca37cddb90db8d1228561db370ca58"
server_key_version: "encryptionkey@0"
I20260926 02:16:49.720309 12145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:16:49.751475 12145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:16:49.755537 12145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:16:49.755776 12145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:16:49.756723 12145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:16:49.758138 12145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:16:49.758246 12145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:49.758327 12145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:16:49.758379 12145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:16:49.788506 12145 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:35381
I20260926 02:16:49.788561 12274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:35381 every 8 connection(s)
I20260926 02:16:49.790047 12145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:16:49.791015 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 12145
I20260926 02:16:49.795418 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:49.793554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:50597)
0926 02:16:49.794301 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:49.794531 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":280,"threads_started":1}
W20260926 02:16:49.796069 12275 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:45565 (0 consecutive failures): Network error: Failed to ping master at 127.28.152.190:45565: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
I20260926 02:16:49.796934 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:49.796538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:48235)
0926 02:16:49.796698 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:49.796821 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 02:16:49.797794 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:49.797470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:50091)
0926 02:16:49.797608 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:49.797688 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 02:16:49.798029 12275 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:16:49.883755 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:49.882677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:41165)
0926 02:16:49.883273 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:49.883378 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":306,"threads_started":1}
I20260926 02:16:50.339732 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:50.338893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:38727)
0926 02:16:50.339369 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:50.339488 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":232,"threads_started":1}
I20260926 02:16:50.799618 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:50.798757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:47977)
0926 02:16:50.799245 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:50.799378 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":228,"threads_started":1}
I20260926 02:16:50.885589 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:50.884793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:45797)
0926 02:16:50.885321 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:50.885413 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":257,"threads_started":1}
I20260926 02:16:51.341683 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:51.340837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:55803)
0926 02:16:51.341386 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:51.341481 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":287,"threads_started":1}
I20260926 02:16:51.801354 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:51.800575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:54113)
0926 02:16:51.801047 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:51.801187 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":246,"threads_started":1}
I20260926 02:16:51.887356 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:51.886553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:42363)
0926 02:16:51.887054 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:51.887153 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":240,"threads_started":1}
I20260926 02:16:52.343403 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:52.342640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:43653)
0926 02:16:52.343112 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:52.343190 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":231,"threads_started":1}
I20260926 02:16:52.803176 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:52.802372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:41527)
0926 02:16:52.802863 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:52.802996 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":230,"threads_started":1}
I20260926 02:16:52.889067 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:52.888319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:60007)
0926 02:16:52.888802 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:52.888898 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":247,"threads_started":1}
I20260926 02:16:53.345265 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:53.344428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:57305)
0926 02:16:53.344916 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:53.345006 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":242,"threads_started":1}
I20260926 02:16:53.804770 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:53.804054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:41149)
0926 02:16:53.804509 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:53.804593 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":220,"threads_started":1}
I20260926 02:16:53.890789 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:53.890037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:41761)
0926 02:16:53.890511 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:53.890611 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":238,"threads_started":1}
I20260926 02:16:54.346979 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:54.346250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:47299)
0926 02:16:54.346725 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:54.346810 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":233,"threads_started":1}
I20260926 02:16:54.806579 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:54.805813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:54985)
0926 02:16:54.806307 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:54.806409 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":248,"threads_started":1}
I20260926 02:16:54.892390 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:54.891671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:44247)
0926 02:16:54.892136 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:54.892231 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":238,"threads_started":1}
I20260926 02:16:55.348707 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:55.347931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:34493)
0926 02:16:55.348443 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:55.348544 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":258,"threads_started":1}
I20260926 02:16:55.808301 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:55.807572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:46075)
0926 02:16:55.808024 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:55.808127 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
I20260926 02:16:55.894398 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:55.893548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:60681)
0926 02:16:55.894091 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:55.894190 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":246,"threads_started":1}
I20260926 02:16:56.350538 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:56.349723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:34567)
0926 02:16:56.350231 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:56.350320 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":226,"threads_started":1}
I20260926 02:16:56.810307 12299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:56.809377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:58441)
0926 02:16:56.809927 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:56.810024 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":234,"threads_started":1}
I20260926 02:16:56.896175 12300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:56.895420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:58797)
0926 02:16:56.895916 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:56.896006 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":267,"threads_started":1}
I20260926 02:16:57.352317 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:57.351555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:37575)
0926 02:16:57.352051 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:57.352143 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":252,"threads_started":1}
I20260926 02:16:57.812039 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:57.811319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:58301)
0926 02:16:57.811781 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:57.811878 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":230,"threads_started":1}
I20260926 02:16:57.898120 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:57.897213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:38015)
0926 02:16:57.897779 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:57.897892 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":268,"threads_started":1}
I20260926 02:16:58.354105 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:58.353336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:46173)
0926 02:16:58.353839 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:58.353931 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":225,"threads_started":1}
I20260926 02:16:58.814026 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:58.813073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:33089)
0926 02:16:58.813695 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:58.813836 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":328,"threads_started":1}
I20260926 02:16:58.899945 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:58.899220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:44027)
0926 02:16:58.899676 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:58.899768 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
I20260926 02:16:59.356073 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:59.355187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:44911)
0926 02:16:59.355797 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:59.355896 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":323,"threads_started":1}
I20260926 02:16:59.816018 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:59.815196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:39899)
0926 02:16:59.815745 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:59.815845 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":275,"threads_started":1}
I20260926 02:16:59.901921 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:16:59.901080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:56561)
0926 02:16:59.901627 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 02:16:59.901722 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":323,"threads_started":1}
I20260926 02:17:00.358034 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:00.357119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:53071)
0926 02:17:00.357694 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:00.357845 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":249,"threads_started":1}
I20260926 02:17:00.817693 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:00.816982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:47085)
0926 02:17:00.817433 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:00.817519 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
I20260926 02:17:00.903909 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:00.903053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:39483)
0926 02:17:00.903592 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:00.903693 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":306,"threads_started":1}
I20260926 02:17:01.360024 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:01.359233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:56923)
0926 02:17:01.359751 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:01.359853 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":256,"threads_started":1}
I20260926 02:17:01.819484 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:01.818754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:52299)
0926 02:17:01.819222 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:01.819316 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":238,"threads_started":1}
I20260926 02:17:01.905993 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:01.905041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:35771)
0926 02:17:01.905627 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:01.905743 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":340,"threads_started":1}
I20260926 02:17:02.361860 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:02.361013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:60743)
0926 02:17:02.361568 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:02.361667 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":292,"threads_started":1}
I20260926 02:17:02.821391 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:02.820532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:42819)
0926 02:17:02.821049 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:02.821186 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":239,"threads_started":1}
I20260926 02:17:02.907706 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:02.906947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:33171)
0926 02:17:02.907423 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:02.907524 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":241,"threads_started":1}
I20260926 02:17:03.363713 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:03.362882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:51717)
0926 02:17:03.363418 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:03.363517 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":266,"threads_started":1}
I20260926 02:17:03.823058 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:03.822298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:47845)
0926 02:17:03.822760 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:03.822850 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":237,"threads_started":1}
I20260926 02:17:03.909862 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:03.908994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.129:34365)
0926 02:17:03.909546 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:03.909654 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":298,"threads_started":1}
I20260926 02:17:04.365779 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:04.364993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.130:44891)
0926 02:17:04.365456 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:04.365575 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":248,"threads_started":1}
I20260926 02:17:04.796689 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 11844
I20260926 02:17:04.816648 12324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:04.797968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.28.152.129:46827)
0926 02:17:04.798588 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:04.798607 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:04.798698 (+ 91us) server_negotiation.cc:407] Connection header received
0926 02:17:04.800506 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:04.800518 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:04.801004 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:04.801438 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:04.803139 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:04.804619 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:04.806146 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:04.806921 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:04.807087 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:04.810074 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:04.810083 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:04.810102 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:04.810158 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:04.813327 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:04.814110 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:04.814118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:04.814126 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:04.814312 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:04.814653 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:04.814656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:04.814658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:04.815295 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:04.815537 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:04.815862 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 02:17:04.816343 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":264,"threads_started":1}
I20260926 02:17:04.818230 12004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:17:04.825246 12326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:04.824366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45565 (local address 127.28.152.131:59249)
0926 02:17:04.824907 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:04.825016 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
I20260926 02:17:04.904938 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 11844
I20260926 02:17:04.919943 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 12012
I20260926 02:17:04.939077 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:04.920973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.28.152.130:45943)
0926 02:17:04.921546 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:04.921560 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:04.921599 (+ 39us) server_negotiation.cc:407] Connection header received
0926 02:17:04.923182 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:04.923193 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:04.923660 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:04.923969 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:04.925635 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:04.927033 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:04.928370 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:04.929119 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:04.929313 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:04.932221 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:04.932230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:04.932249 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:04.932297 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:04.935409 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:04.936279 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:04.936288 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:04.936296 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:04.936521 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:04.936940 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:04.936942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:04.936944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:04.937641 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:04.937881 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:04.938374 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 02:17:04.938778 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":264,"threads_started":1}
I20260926 02:17:04.940554 12137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:17:05.024334 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 12012
I20260926 02:17:05.038632 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 12145
I20260926 02:17:05.056674 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:05.039587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.28.152.131:35381)
0926 02:17:05.040172 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:05.040186 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:05.040229 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:17:05.041942 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:05.041953 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:05.042407 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:05.042724 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:05.044085 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:05.045439 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:05.046749 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:05.047503 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:05.047658 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:05.050382 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:05.050391 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:05.050410 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:05.050459 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:05.053451 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:05.054176 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:05.054184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:05.054191 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:05.054364 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:05.054769 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:05.054772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:05.054774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:05.055325 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:05.055592 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:05.055987 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 02:17:05.056340 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":269,"threads_started":1}
I20260926 02:17:05.058151 12270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:17:05.141096 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 12145
2026-09-26T02:17:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24406 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:17:05 dist-test-slave-hllj krb5kdc[12333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:17:05 dist-test-slave-hllj krb5kdc[12333](info): set up 2 sockets
Sep 26 02:17:05 dist-test-slave-hllj krb5kdc[12333](info): commencing operation
krb5kdc: starting...
W20260926 02:17:06.162806 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:17:06 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:17:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:17:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:17:06.401429 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34341
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:17:06.737624 12346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:06.737958 12346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:06.738097 12346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:06.745580 12346 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:17:06.745708 12346 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:17:06.745769 12346 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:17:06.745837 12346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:17:06.745921 12346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:06.746016 12346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:06.746080 12346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:17:06.746155 12346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:17:06.756491 12346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:06.757907 12346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:06.759313 12346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:06.771903 12352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:06.771899 12354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:06.772046 12351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:06.772423 12346 server_base.cc:1034] running on GCE node
I20260926 02:17:06.773564 12346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:06.775126 12346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:06.776396 12346 hybrid_clock.cc:648] HybridClock initialized: now 1790389026776319 us; error 50 us; skew 500 ppm
I20260926 02:17:06.776786 12346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:06 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:06.783939 12346 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:17:06.785671 12346 webserver.cc:460] Webserver started at http://127.28.152.190:40481/ using document root <none> and password file <none>
I20260926 02:17:06.786410 12346 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:06.786514 12346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:06.786829 12346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:17:06.790100 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c"
format_stamp: "Formatted at 2026-09-26 02:17:06 on dist-test-slave-hllj"
server_key: "cbb0996bae36f2dd3a3f2b7c8b6bd57a"
server_key_iv: "0c388916738c9f5257cc87b20a4b299b"
server_key_version: "encryptionkey@0"
I20260926 02:17:06.790936 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c"
format_stamp: "Formatted at 2026-09-26 02:17:06 on dist-test-slave-hllj"
server_key: "cbb0996bae36f2dd3a3f2b7c8b6bd57a"
server_key_iv: "0c388916738c9f5257cc87b20a4b299b"
server_key_version: "encryptionkey@0"
I20260926 02:17:06.796765 12346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260926 02:17:06.800523 12362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:06.802276 12346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:17:06.802469 12346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c"
format_stamp: "Formatted at 2026-09-26 02:17:06 on dist-test-slave-hllj"
server_key: "cbb0996bae36f2dd3a3f2b7c8b6bd57a"
server_key_iv: "0c388916738c9f5257cc87b20a4b299b"
server_key_version: "encryptionkey@0"
I20260926 02:17:06.802722 12346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:06.819201 12346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:06.824069 12346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:06.824348 12346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:06.842283 12346 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:34341
I20260926 02:17:06.842290 12414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:34341 every 8 connection(s)
I20260926 02:17:06.843803 12346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:17:06.844560 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 12346
I20260926 02:17:06.844794 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:17:06.845584 29282 external_mini_cluster.cc:1428] Setting key e19ab341841cd8f710150156a141ff50
I20260926 02:17:06.848744 12415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:17:06 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:17:06.862743 12415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Bootstrap starting.
I20260926 02:17:06.866962 12415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Neither blocks nor log segments found. Creating new log.
I20260926 02:17:06.868299 12415 log.cc:824] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:17:06.872211 12415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: No bootstrap required, opened a new log
I20260926 02:17:06.872841 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:06.847682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.28.152.190:34341)
0926 02:17:06.848517 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:06.848534 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:06.848585 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:17:06.850971 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:06.851002 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:06.851552 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:06.852053 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:06.853530 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:06.855729 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:06.857500 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:06.858625 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:06.858837 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:06.863551 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:06.863564 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:06.863586 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:06.863643 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:06.867655 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:06.868504 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:06.868515 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:06.868525 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:06.868748 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:06.869306 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:06.869311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:06.869313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:06.870029 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:06.870335 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:06.870771 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 02:17:06.871330 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":324,"threads_started":1}
I20260926 02:17:06.880272 12415 raft_consensus.cc:348] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:06.880676 12415 raft_consensus.cc:374] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:17:06.880751 12415 raft_consensus.cc:729] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32e7fca9e3b4bb9ae0150cda876fe6c, State: Initialized, Role: FOLLOWER
I20260926 02:17:06.881333 12415 consensus_queue.cc:260] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:06.881626 12415 raft_consensus.cc:388] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:17:06.881788 12415 raft_consensus.cc:482] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:17:06.882001 12415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:17:06.884406 12415 raft_consensus.cc:504] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:06.885001 12415 leader_election.cc:302] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f32e7fca9e3b4bb9ae0150cda876fe6c; no voters:
I20260926 02:17:06.885679 12415 leader_election.cc:288] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:17:06.885820 12420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:17:06.886965 12420 raft_consensus.cc:686] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 1 LEADER]: Becoming Leader. State: Replica: f32e7fca9e3b4bb9ae0150cda876fe6c, State: Running, Role: LEADER
I20260926 02:17:06.887575 12420 consensus_queue.cc:237] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:06.887984 12415 sys_catalog.cc:564] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:17:06.890228 12422 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f32e7fca9e3b4bb9ae0150cda876fe6c. Latest consensus state: current_term: 1 leader_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } } }
I20260926 02:17:06.890313 12421 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } } }
I20260926 02:17:06.890614 12422 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: This master's current role is: LEADER
I20260926 02:17:06.890666 12421 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: This master's current role is: LEADER
I20260926 02:17:06.894520 12427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:17:06.900081 12427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:17:06.909948 12427 catalog_manager.cc:1347] Generated new cluster ID: d23bd841c6004e3582609bef4bec1293
I20260926 02:17:06.910074 12427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:17:06.940275 12427 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:17:06.942395 12427 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:17:06.952625 12427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Generated new TSK 0
I20260926 02:17:06.953876 12427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:17:07.028947 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:17:07.367515 12441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:07.367841 12441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:07.367980 12441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:07.375623 12441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:07.375761 12441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:07.375928 12441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:17:07.386042 12441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:07.387387 12441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:07.388814 12441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:07.402141 12447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:07.402196 12448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:07.402238 12450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:07.403156 12441 server_base.cc:1034] running on GCE node
I20260926 02:17:07.403808 12441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:07.404781 12441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:07.405972 12441 hybrid_clock.cc:648] HybridClock initialized: now 1790389027405907 us; error 60 us; skew 500 ppm
I20260926 02:17:07.406291 12441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:07 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:07.409442 12441 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:17:07.410933 12441 webserver.cc:460] Webserver started at http://127.28.152.129:38991/ using document root <none> and password file <none>
I20260926 02:17:07.411640 12441 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:07.411741 12441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:07.411996 12441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:17:07.415056 12441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "2a6b55930f1f49e1b1f120231d0bf80e"
format_stamp: "Formatted at 2026-09-26 02:17:07 on dist-test-slave-hllj"
server_key: "fdd509cce9fb292586503671becbd31a"
server_key_iv: "4027060e2f02faf573620d3eeb2e31ff"
server_key_version: "encryptionkey@0"
I20260926 02:17:07.415798 12441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "2a6b55930f1f49e1b1f120231d0bf80e"
format_stamp: "Formatted at 2026-09-26 02:17:07 on dist-test-slave-hllj"
server_key: "fdd509cce9fb292586503671becbd31a"
server_key_iv: "4027060e2f02faf573620d3eeb2e31ff"
server_key_version: "encryptionkey@0"
I20260926 02:17:07.421407 12441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:17:07.425283 12458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:07.426854 12441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:17:07.427029 12441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "2a6b55930f1f49e1b1f120231d0bf80e"
format_stamp: "Formatted at 2026-09-26 02:17:07 on dist-test-slave-hllj"
server_key: "fdd509cce9fb292586503671becbd31a"
server_key_iv: "4027060e2f02faf573620d3eeb2e31ff"
server_key_version: "encryptionkey@0"
I20260926 02:17:07.427235 12441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:07.446100 12441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:07.450551 12441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:07.450822 12441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:07.451964 12441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:17:07.453449 12441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:17:07.453528 12441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:07.453651 12441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:17:07.453703 12441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:07.491250 12441 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:42041
I20260926 02:17:07.491290 12571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:42041 every 8 connection(s)
I20260926 02:17:07.492559 12441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:17:07.495532 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 12441
I20260926 02:17:07.495767 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:17:07.496397 29282 external_mini_cluster.cc:1428] Setting key d7ff23e6c3d1030fac7a1c5b94e1f930
Sep 26 02:17:07 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:17:07.515439 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:07.495360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:58029 (local address 127.28.152.190:34341)
0926 02:17:07.495854 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:07.495861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:07.497147 (+ 1286us) server_negotiation.cc:407] Connection header received
0926 02:17:07.499364 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:07.499368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:07.499434 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:07.499522 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:07.501679 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:07.502421 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:07.504692 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:07.505102 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:07.505248 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:07.509416 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:07.509421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:07.509444 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:07.509484 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:07.512007 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:07.512960 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:07.512965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:07.512967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:07.513107 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:07.513612 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:07.513616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:07.513618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:07.513912 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:07.514042 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:07.514977 (+ 935us) server_negotiation.cc:299] Negotiation successful
0926 02:17:07.515169 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":309,"threads_started":1}
I20260926 02:17:07.516755 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:07.496056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:58029)
0926 02:17:07.496935 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:07.496990 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:07.499029 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:07.499821 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:07.499835 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:07.500607 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:17:07.501479 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:07.501506 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:07.502586 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:07.502591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:07.504525 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:07.504541 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:07.505359 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:07.505366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:07.505661 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:07.506429 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:17:07.506465 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:17:07.509157 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:17:07.512154 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:07.512165 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:07.512183 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:07.512793 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:07.513245 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:07.513250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:07.513254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:07.513503 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:07.514038 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:17:07.514046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:17:07.514489 (+ 443us) client_negotiation.cc:769] Sending connection context
0926 02:17:07.514810 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 02:17:07.515358 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":378,"threads_started":1}
I20260926 02:17:07.518412 12572 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34341
I20260926 02:17:07.518718 12572 heartbeater.cc:464] Registering TS with master...
I20260926 02:17:07.519471 12572 heartbeater.cc:511] Master 127.28.152.190:34341 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
I20260926 02:17:07.521600 12379 ts_manager.cc:194] Registered new tserver with Master: 2a6b55930f1f49e1b1f120231d0bf80e (127.28.152.129:42041)
I20260926 02:17:07.523151 12379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:58029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:17:07.552060 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:17:07.882735 12578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:07.883033 12578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:07.883140 12578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:07.890586 12578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:07.890710 12578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:07.890836 12578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:17:07.900775 12578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:07.902190 12578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:07.903580 12578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:07.917716 12584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:07.917672 12586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:07.917631 12583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:07.922896 12578 server_base.cc:1034] running on GCE node
I20260926 02:17:07.923681 12578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:07.924799 12578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:07.926009 12578 hybrid_clock.cc:648] HybridClock initialized: now 1790389027925933 us; error 73 us; skew 500 ppm
I20260926 02:17:07.926379 12578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:07 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:07.929704 12578 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:17:07.931486 12578 webserver.cc:460] Webserver started at http://127.28.152.130:44909/ using document root <none> and password file <none>
I20260926 02:17:07.932287 12578 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:07.932417 12578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:07.932705 12578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:17:07.935850 12578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "0d9a4ee5082042d58641ed5ec8b980f3"
format_stamp: "Formatted at 2026-09-26 02:17:07 on dist-test-slave-hllj"
server_key: "32843680e736cd7cdac60a2070a3257b"
server_key_iv: "b3489a57f4bd7fde63fb5b846ea3137f"
server_key_version: "encryptionkey@0"
I20260926 02:17:07.936656 12578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "0d9a4ee5082042d58641ed5ec8b980f3"
format_stamp: "Formatted at 2026-09-26 02:17:07 on dist-test-slave-hllj"
server_key: "32843680e736cd7cdac60a2070a3257b"
server_key_iv: "b3489a57f4bd7fde63fb5b846ea3137f"
server_key_version: "encryptionkey@0"
I20260926 02:17:07.942549 12578 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 02:17:07.946395 12593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:07.947960 12578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:17:07.948143 12578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "0d9a4ee5082042d58641ed5ec8b980f3"
format_stamp: "Formatted at 2026-09-26 02:17:07 on dist-test-slave-hllj"
server_key: "32843680e736cd7cdac60a2070a3257b"
server_key_iv: "b3489a57f4bd7fde63fb5b846ea3137f"
server_key_version: "encryptionkey@0"
I20260926 02:17:07.948328 12578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:07.974145 12578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:07.978499 12578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:07.978760 12578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:07.979753 12578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:17:07.981345 12578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:17:07.981437 12578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:07.981575 12578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:17:07.981637 12578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:08.012605 12578 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:33409
I20260926 02:17:08.012692 12706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:33409 every 8 connection(s)
I20260926 02:17:08.013899 12578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:17:08.015214 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 12578
I20260926 02:17:08.015398 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:17:08.015836 29282 external_mini_cluster.cc:1428] Setting key 18ae1caacd1ce756f0ec200a5a890f51
Sep 26 02:17:08 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:17:08.035502 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.016475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:34645 (local address 127.28.152.190:34341)
0926 02:17:08.016905 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:08.016911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:08.018165 (+ 1254us) server_negotiation.cc:407] Connection header received
0926 02:17:08.020061 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:08.020067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:08.020146 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:08.020240 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:08.021923 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.022718 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.024355 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.024793 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.024904 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.028820 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:08.028827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:08.028830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:08.028869 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:08.031720 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:08.032799 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:08.032805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:08.032807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:08.032969 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:08.033537 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:08.033541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:08.033543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:08.033839 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:08.034048 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:08.034858 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 02:17:08.035122 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260926 02:17:08.036595 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.017210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:34645)
0926 02:17:08.018003 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:08.018064 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:08.019773 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:08.020510 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:08.020519 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:08.021075 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:17:08.021745 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.021766 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.022879 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.022883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.024203 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.024216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.025027 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.025035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.025315 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.025983 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:17:08.026010 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:17:08.028496 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:17:08.031898 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:08.031909 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:08.031930 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:08.032626 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:08.033123 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:08.033152 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:08.033155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:08.033428 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:08.033991 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:17:08.034000 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:17:08.034426 (+ 426us) client_negotiation.cc:769] Sending connection context
0926 02:17:08.034721 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 02:17:08.035319 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":280,"threads_started":1}
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:17:08.038290 12707 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34341
I20260926 02:17:08.038666 12707 heartbeater.cc:464] Registering TS with master...
I20260926 02:17:08.039448 12707 heartbeater.cc:511] Master 127.28.152.190:34341 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
I20260926 02:17:08.041401 12379 ts_manager.cc:194] Registered new tserver with Master: 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409)
I20260926 02:17:08.042258 12379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:34645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:17:08.068634 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:17:08.403895 12713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:08.404212 12713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:08.404309 12713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:08.411919 12713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:08.412055 12713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:08.412169 12713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:17:08.422209 12713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:08.423578 12713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:08.424976 12713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:08.438604 12721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:08.438648 12718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:08.438666 12719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:08.439465 12713 server_base.cc:1034] running on GCE node
I20260926 02:17:08.440160 12713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:08.441211 12713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:08.442427 12713 hybrid_clock.cc:648] HybridClock initialized: now 1790389028442379 us; error 42 us; skew 500 ppm
I20260926 02:17:08.442754 12713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:08 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389028, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:08.446060 12713 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:17:08.447578 12713 webserver.cc:460] Webserver started at http://127.28.152.131:42659/ using document root <none> and password file <none>
I20260926 02:17:08.448307 12713 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:08.448436 12713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:08.448751 12713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:17:08.452102 12713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "4c46720e86fd46c988754878e59a9d61"
format_stamp: "Formatted at 2026-09-26 02:17:08 on dist-test-slave-hllj"
server_key: "08622747c0fed1b7dca61b42d8c233b0"
server_key_iv: "d6616eb1c0c23131bc1a50b9deeddf7e"
server_key_version: "encryptionkey@0"
I20260926 02:17:08.452924 12713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "4c46720e86fd46c988754878e59a9d61"
format_stamp: "Formatted at 2026-09-26 02:17:08 on dist-test-slave-hllj"
server_key: "08622747c0fed1b7dca61b42d8c233b0"
server_key_iv: "d6616eb1c0c23131bc1a50b9deeddf7e"
server_key_version: "encryptionkey@0"
I20260926 02:17:08.458490 12713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:17:08.463052 12728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:08.464583 12713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:17:08.464759 12713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "4c46720e86fd46c988754878e59a9d61"
format_stamp: "Formatted at 2026-09-26 02:17:08 on dist-test-slave-hllj"
server_key: "08622747c0fed1b7dca61b42d8c233b0"
server_key_iv: "d6616eb1c0c23131bc1a50b9deeddf7e"
server_key_version: "encryptionkey@0"
I20260926 02:17:08.464969 12713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:08.481740 12713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:08.485806 12713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:08.486088 12713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:08.487115 12713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:17:08.488504 12713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:17:08.488582 12713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:08.488703 12713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:17:08.488756 12713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:08.519699 12713 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:43359
I20260926 02:17:08.519734 12841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:43359 every 8 connection(s)
I20260926 02:17:08.520982 12713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:17:08.521476 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 12713
I20260926 02:17:08.521704 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:17:08.522312 29282 external_mini_cluster.cc:1428] Setting key 22480d6dead4fb9df68c3168f2e8199a
I20260926 02:17:08.526250 12572 heartbeater.cc:503] Master 127.28.152.190:34341 was elected leader, sending a full tablet report...
Sep 26 02:17:08 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389028, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:17:08.543112 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.524245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:44841 (local address 127.28.152.190:34341)
0926 02:17:08.524477 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:08.524483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:08.525859 (+ 1376us) server_negotiation.cc:407] Connection header received
0926 02:17:08.528200 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:08.528205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:08.528305 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:08.528387 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:08.530162 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.530900 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.532638 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.533043 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.533186 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.537292 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:08.537299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:08.537303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:08.537348 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:08.540092 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:08.540891 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:08.540895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:08.540898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:08.541026 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:08.541495 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:08.541498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:08.541500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:08.541797 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:08.541916 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:08.542665 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 02:17:08.542857 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:17:08.544168 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.524847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:44841)
0926 02:17:08.525715 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:08.525755 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:08.527906 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:08.528670 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:08.528679 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:08.529320 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:17:08.529985 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.530006 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.531090 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.531094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.532486 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.532501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.533297 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.533305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.533561 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.534291 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:17:08.534318 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:17:08.536949 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:17:08.540251 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:08.540261 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:08.540276 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:08.540786 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:08.541173 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:08.541178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:08.541182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:08.541400 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:08.541915 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:17:08.541922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:17:08.542307 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 02:17:08.542582 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 02:17:08.543020 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":339,"threads_started":1}
I20260926 02:17:08.545912 12842 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34341
I20260926 02:17:08.546234 12842 heartbeater.cc:464] Registering TS with master...
I20260926 02:17:08.546969 12842 heartbeater.cc:511] Master 127.28.152.190:34341 requested a full tablet report, sending...
I20260926 02:17:08.548557 12379 ts_manager.cc:194] Registered new tserver with Master: 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359)
I20260926 02:17:08.549533 12379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:44841
I20260926 02:17:08.551438 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:17:08.569588 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.556203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.28.152.190:34341)
0926 02:17:08.556421 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:08.556428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:08.556804 (+ 376us) server_negotiation.cc:407] Connection header received
0926 02:17:08.556956 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:08.556960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:08.557038 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:08.557170 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:08.558596 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.559371 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.560854 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.561318 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.561414 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.563895 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:08.563901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:08.563904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:08.563941 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:08.566617 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:08.567307 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:08.567311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:08.567313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:08.567445 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:08.567885 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:08.567889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:08.567892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:08.568309 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:08.568554 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:08.568940 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 02:17:08.569219 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:17:08.574178 12379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:17:08.581027 12379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:17:08.619258 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.596124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.28.152.130:33409)
0926 02:17:08.610004 (+ 13880us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:08.610014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:08.611804 (+ 1790us) server_negotiation.cc:407] Connection header received
0926 02:17:08.612115 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:08.612124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:08.612430 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:08.612613 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:17:08.613510 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.614614 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.616784 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.617916 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.618086 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.618288 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:08.618711 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 02:17:08.618909 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13702,"thread_start_us":13591,"threads_started":1}
I20260926 02:17:08.619367 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.595853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33409 (local address 127.0.0.1:53742)
0926 02:17:08.611689 (+ 15836us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:08.611707 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:08.611914 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:08.612897 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:08.612902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:08.612931 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:17:08.613296 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.613308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.614746 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.614750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.616595 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.616609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.618131 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.618138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.618448 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.618499 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 02:17:08.618631 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 02:17:08.618894 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15699,"mutex_wait_us":62,"thread_start_us":15510,"threads_started":1}
I20260926 02:17:08.620201 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.594487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.28.152.131:43359)
0926 02:17:08.610439 (+ 15952us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:08.610445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:08.610473 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:17:08.610565 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:08.610571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:08.610785 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:08.610990 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:17:08.611862 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.613469 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.616548 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.618058 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.619386 (+ 1328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.619580 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:08.619666 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 02:17:08.619836 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15806,"thread_start_us":15646,"threads_started":1}
I20260926 02:17:08.620708 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.593913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:43359 (local address 127.0.0.1:48580)
0926 02:17:08.607810 (+ 13897us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:08.607841 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:08.608052 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:08.611052 (+ 3000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:08.611062 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:08.611101 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:17:08.611614 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.611629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.613659 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.613665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.616322 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.616340 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.618324 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.618330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.618553 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.618592 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:17:08.618667 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 02:17:08.618897 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13697,"mutex_wait_us":60,"thread_start_us":959,"threads_started":1}
I20260926 02:17:08.621867 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.593507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42041 (local address 127.0.0.1:52734)
0926 02:17:08.594583 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:08.594628 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:08.594871 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:08.614297 (+ 19426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:08.614303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:08.614330 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:17:08.614777 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.614789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.616937 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.616943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.618961 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.618975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.621050 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.621056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.621257 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.621295 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:17:08.621367 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 02:17:08.621523 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":1,"thread_start_us":14655,"threads_started":1}
I20260926 02:17:08.622583 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.593834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.28.152.129:42041)
0926 02:17:08.613637 (+ 19803us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:08.613646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:08.613676 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:17:08.613776 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:08.613784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:08.614038 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:08.615003 (+ 965us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:17:08.615283 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.616775 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.619168 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.620798 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.621825 (+ 1027us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.622009 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:08.622080 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 02:17:08.622249 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19633,"thread_start_us":19482,"threads_started":1}
I20260926 02:17:08.624337 12641 tablet_service.cc:1467] Processing CreateTablet for tablet 083c374218804b7497fe1516119a48ed (DEFAULT_TABLE table=test-table [id=4c833a6719214e359c6df89a9a344850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:17:08.624724 12776 tablet_service.cc:1467] Processing CreateTablet for tablet 083c374218804b7497fe1516119a48ed (DEFAULT_TABLE table=test-table [id=4c833a6719214e359c6df89a9a344850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:17:08.625784 12641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083c374218804b7497fe1516119a48ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:17:08.626748 12776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083c374218804b7497fe1516119a48ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:17:08.626520 12506 tablet_service.cc:1467] Processing CreateTablet for tablet 083c374218804b7497fe1516119a48ed (DEFAULT_TABLE table=test-table [id=4c833a6719214e359c6df89a9a344850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:17:08.628007 12506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083c374218804b7497fe1516119a48ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:17:08.637993 12860 tablet_bootstrap.cc:492] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: Bootstrap starting.
I20260926 02:17:08.641798 12860 tablet_bootstrap.cc:654] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: Neither blocks nor log segments found. Creating new log.
I20260926 02:17:08.642860 12860 log.cc:824] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:17:08.643823 12861 tablet_bootstrap.cc:492] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: Bootstrap starting.
I20260926 02:17:08.644757 12862 tablet_bootstrap.cc:492] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: Bootstrap starting.
I20260926 02:17:08.647260 12860 tablet_bootstrap.cc:492] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: No bootstrap required, opened a new log
I20260926 02:17:08.647541 12860 ts_tablet_manager.cc:1397] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 02:17:08.648555 12861 tablet_bootstrap.cc:654] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: Neither blocks nor log segments found. Creating new log.
I20260926 02:17:08.648562 12862 tablet_bootstrap.cc:654] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: Neither blocks nor log segments found. Creating new log.
I20260926 02:17:08.649941 12861 log.cc:824] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:17:08.650257 12862 log.cc:824] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: Log is configured to *not* fsync() on all Append() calls
I20260926 02:17:08.653283 12861 tablet_bootstrap.cc:492] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: No bootstrap required, opened a new log
I20260926 02:17:08.653585 12862 tablet_bootstrap.cc:492] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: No bootstrap required, opened a new log
I20260926 02:17:08.653564 12861 ts_tablet_manager.cc:1397] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:17:08.653909 12862 ts_tablet_manager.cc:1397] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 02:17:08.657706 12860 raft_consensus.cc:348] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:08.658259 12860 raft_consensus.cc:374] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:17:08.658420 12860 raft_consensus.cc:729] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9a4ee5082042d58641ed5ec8b980f3, State: Initialized, Role: FOLLOWER
I20260926 02:17:08.659160 12860 consensus_queue.cc:260] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:08.661219 12860 ts_tablet_manager.cc:1428] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260926 02:17:08.661794 12707 heartbeater.cc:503] Master 127.28.152.190:34341 was elected leader, sending a full tablet report...
I20260926 02:17:08.661950 12861 raft_consensus.cc:348] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:08.662360 12861 raft_consensus.cc:374] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:17:08.662467 12861 raft_consensus.cc:729] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6b55930f1f49e1b1f120231d0bf80e, State: Initialized, Role: FOLLOWER
I20260926 02:17:08.663092 12861 consensus_queue.cc:260] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:08.664669 12862 raft_consensus.cc:348] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:08.665056 12861 ts_tablet_manager.cc:1428] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 02:17:08.665233 12862 raft_consensus.cc:374] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:17:08.665370 12862 raft_consensus.cc:729] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c46720e86fd46c988754878e59a9d61, State: Initialized, Role: FOLLOWER
I20260926 02:17:08.666095 12862 consensus_queue.cc:260] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:08.668393 12842 heartbeater.cc:503] Master 127.28.152.190:34341 was elected leader, sending a full tablet report...
I20260926 02:17:08.669203 12862 ts_tablet_manager.cc:1428] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
W20260926 02:17:08.745203 12573 tablet.cc:2369] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:17:08.760746 12867 raft_consensus.cc:482] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:17:08.761051 12867 raft_consensus.cc:504] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:08.762873 12867 leader_election.cc:288] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359), 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409)
W20260926 02:17:08.765780 12708 tablet.cc:2369] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:17:08.770493 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.763339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:44533 (local address 127.28.152.131:43359)
0926 02:17:08.763596 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:08.763604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:08.764144 (+ 540us) server_negotiation.cc:407] Connection header received
0926 02:17:08.764451 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:08.764457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:08.764547 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:08.764614 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:17:08.765461 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.766661 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.768591 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.769559 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.769706 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.769783 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:08.770150 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 02:17:08.770254 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:17:08.770711 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.763329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:43359 (local address 127.28.152.129:44533)
0926 02:17:08.764025 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:08.764056 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:08.764271 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:08.764822 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:08.764827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:08.764858 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:17:08.765276 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.765290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.766793 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.766797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.768403 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.768423 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.769795 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.769801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.769963 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.769996 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:17:08.770112 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 02:17:08.770273 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":39,"spinlock_wait_cycles":52480,"thread_start_us":384,"threads_started":1}
I20260926 02:17:08.770851 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.764202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33409 (local address 127.28.152.129:52489)
0926 02:17:08.764868 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:08.764885 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:08.765062 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:08.765461 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:08.765465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:08.765492 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:17:08.765886 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.765898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.767203 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.767209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.768696 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:08.768710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.770105 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.770110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:08.770265 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.770292 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 02:17:08.770374 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 02:17:08.770507 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":2,"spinlock_wait_cycles":896,"thread_start_us":351,"threads_started":1}
I20260926 02:17:08.771000 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.764335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:52489 (local address 127.28.152.130:33409)
0926 02:17:08.764615 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:08.764621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:08.764982 (+ 361us) server_negotiation.cc:407] Connection header received
0926 02:17:08.765221 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:08.765227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:08.765299 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:08.765388 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:17:08.766099 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.767070 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.768883 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.769871 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.770052 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.770124 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:08.770408 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 02:17:08.770546 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260926 02:17:08.771831 12796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083c374218804b7497fe1516119a48ed" candidate_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c46720e86fd46c988754878e59a9d61" is_pre_election: true
I20260926 02:17:08.771837 12661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083c374218804b7497fe1516119a48ed" candidate_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" is_pre_election: true
I20260926 02:17:08.772322 12796 raft_consensus.cc:2454] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a6b55930f1f49e1b1f120231d0bf80e in term 0.
I20260926 02:17:08.772320 12661 raft_consensus.cc:2454] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a6b55930f1f49e1b1f120231d0bf80e in term 0.
W20260926 02:17:08.772823 12843 tablet.cc:2369] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:17:08.773046 12459 leader_election.cc:302] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a6b55930f1f49e1b1f120231d0bf80e, 4c46720e86fd46c988754878e59a9d61; no voters:
I20260926 02:17:08.773553 12867 raft_consensus.cc:2781] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:17:08.773705 12867 raft_consensus.cc:482] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:17:08.773826 12867 raft_consensus.cc:3037] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:17:08.776067 12867 raft_consensus.cc:504] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:08.776741 12867 leader_election.cc:288] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 1 election: Requested vote from peers 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359), 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409)
I20260926 02:17:08.777374 12796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083c374218804b7497fe1516119a48ed" candidate_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c46720e86fd46c988754878e59a9d61"
I20260926 02:17:08.777467 12661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083c374218804b7497fe1516119a48ed" candidate_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9a4ee5082042d58641ed5ec8b980f3"
I20260926 02:17:08.777606 12796 raft_consensus.cc:3037] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:17:08.777676 12661 raft_consensus.cc:3037] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:17:08.779882 12661 raft_consensus.cc:2454] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a6b55930f1f49e1b1f120231d0bf80e in term 1.
I20260926 02:17:08.779886 12796 raft_consensus.cc:2454] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a6b55930f1f49e1b1f120231d0bf80e in term 1.
I20260926 02:17:08.780362 12459 leader_election.cc:302] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9a4ee5082042d58641ed5ec8b980f3, 2a6b55930f1f49e1b1f120231d0bf80e; no voters:
I20260926 02:17:08.780704 12867 raft_consensus.cc:2781] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:17:08.781303 12867 raft_consensus.cc:686] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 LEADER]: Becoming Leader. State: Replica: 2a6b55930f1f49e1b1f120231d0bf80e, State: Running, Role: LEADER
I20260926 02:17:08.781814 12867 consensus_queue.cc:237] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:08.786810 12378 catalog_manager.cc:5495] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a6b55930f1f49e1b1f120231d0bf80e (127.28.152.129). New cstate: current_term: 1 leader_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } health_report { overall_health: UNKNOWN } } }
I20260926 02:17:08.815200 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.809536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.28.152.129:42041)
0926 02:17:08.809757 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:08.809762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:08.809806 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:17:08.810025 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:08.810029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:08.810125 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:08.810263 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:17:08.810606 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.811315 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.812959 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.813431 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.813533 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.814044 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:17:08.814608 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:17:08.814721 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:08.814859 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 02:17:08.814935 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:17:08.824304 12661 raft_consensus.cc:1261] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3 [term 1 FOLLOWER]: Refusing update from remote peer 2a6b55930f1f49e1b1f120231d0bf80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:17:08.824303 12796 raft_consensus.cc:1261] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61 [term 1 FOLLOWER]: Refusing update from remote peer 2a6b55930f1f49e1b1f120231d0bf80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:17:08.825328 12871 consensus_queue.cc:1035] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:17:08.825738 12867 consensus_queue.cc:1035] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:17:08.839419 12875 mvcc.cc:205] Tried to move back new op lower bound from 7333433462053056512 to 7333433461894115328. Current Snapshot: MvccSnapshot[applied={T|T < 7333433462053056512}]
I20260926 02:17:08.858165 12379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:17:08.858465 12379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:17:08.862689 12379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Sending DeleteTablet for 3 replicas of tablet 083c374218804b7497fe1516119a48ed
I20260926 02:17:08.863965 12776 tablet_service.cc:1514] Processing DeleteTablet for tablet 083c374218804b7497fe1516119a48ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:17:08 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:48580
I20260926 02:17:08.863977 12506 tablet_service.cc:1514] Processing DeleteTablet for tablet 083c374218804b7497fe1516119a48ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:17:08 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:52734
I20260926 02:17:08.864725 12641 tablet_service.cc:1514] Processing DeleteTablet for tablet 083c374218804b7497fe1516119a48ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:17:08 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:53742
I20260926 02:17:08.865190 12884 tablet_replica.cc:330] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: stopping tablet replica
I20260926 02:17:08.865373 12885 tablet_replica.cc:330] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: stopping tablet replica
I20260926 02:17:08.865677 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 12441
I20260926 02:17:08.865689 12884 raft_consensus.cc:2229] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:17:08.865756 12885 raft_consensus.cc:2229] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:17:08.866226 12884 raft_consensus.cc:2258] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:17:08.866410 12885 raft_consensus.cc:2258] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:17:08.869314 12885 ts_tablet_manager.cc:1905] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:17:08.874380 12884 ts_tablet_manager.cc:1905] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:17:08.877348 12886 tablet_replica.cc:330] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: stopping tablet replica
I20260926 02:17:08.877836 12886 raft_consensus.cc:2229] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:17:08.878409 12886 raft_consensus.cc:2258] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 02:17:08 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:17:08.881999 12884 ts_tablet_manager.cc:1918] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:17:08.882324 12884 log.cc:1196] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/083c374218804b7497fe1516119a48ed
I20260926 02:17:08.882982 12884 ts_tablet_manager.cc:1939] T 083c374218804b7497fe1516119a48ed P 4c46720e86fd46c988754878e59a9d61: Deleting consensus metadata
I20260926 02:17:08.883924 12885 ts_tablet_manager.cc:1918] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:17:08.884140 12885 log.cc:1196] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/083c374218804b7497fe1516119a48ed
I20260926 02:17:08.884745 12885 ts_tablet_manager.cc:1939] T 083c374218804b7497fe1516119a48ed P 2a6b55930f1f49e1b1f120231d0bf80e: Deleting consensus metadata
I20260926 02:17:08.885456 12363 catalog_manager.cc:4841] TS 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359): tablet 083c374218804b7497fe1516119a48ed (table test-table [id=4c833a6719214e359c6df89a9a344850]) successfully deleted
I20260926 02:17:08.886667 12364 catalog_manager.cc:4841] TS 2a6b55930f1f49e1b1f120231d0bf80e (127.28.152.129:42041): tablet 083c374218804b7497fe1516119a48ed (table test-table [id=4c833a6719214e359c6df89a9a344850]) successfully deleted
I20260926 02:17:08.888005 12886 ts_tablet_manager.cc:1905] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:17:08.889276 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:08.867035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.28.152.129:42041)
0926 02:17:08.867200 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:08.867206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:08.867226 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:17:08.867365 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:08.867370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:08.867449 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:08.867567 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:08.869969 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.871227 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.873424 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:08.874128 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:08.874242 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:08.880680 (+ 6438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:08.880697 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:08.880720 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:08.880784 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:08.885380 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:08.886545 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:08.886556 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:08.886560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:08.886758 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:08.887313 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:08.887318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:08.887322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:08.887832 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:08.888221 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:08.888584 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 02:17:08.888844 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:17:08.890218 12567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:17:08.895323 12886 ts_tablet_manager.cc:1918] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:17:08.895490 12886 log.cc:1196] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/083c374218804b7497fe1516119a48ed
I20260926 02:17:08.895967 12886 ts_tablet_manager.cc:1939] T 083c374218804b7497fe1516119a48ed P 0d9a4ee5082042d58641ed5ec8b980f3: Deleting consensus metadata
I20260926 02:17:08.897725 12363 catalog_manager.cc:4841] TS 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409): tablet 083c374218804b7497fe1516119a48ed (table test-table [id=4c833a6719214e359c6df89a9a344850]) successfully deleted
I20260926 02:17:08.988615 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 12441
I20260926 02:17:09.003523 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 12578
Sep 26 02:17:09 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:17:09.020529 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:09.004424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.28.152.130:33409)
0926 02:17:09.004601 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:09.004605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:09.004625 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:17:09.004707 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:09.004711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:09.004774 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:09.004858 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:09.006243 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:09.007134 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:09.008473 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:09.008886 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:09.008985 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:09.013603 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:09.013619 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:09.013643 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:09.013706 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:09.017445 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:09.018239 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:09.018249 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:09.018253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:09.018410 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:09.018774 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:09.018779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:09.018782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:09.019277 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:09.019606 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:09.019885 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 02:17:09.020163 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:17:09.021317 12702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:17:09.113116 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 12578
I20260926 02:17:09.127549 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 12713
Sep 26 02:17:09 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:17:09.143493 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:09.128384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.28.152.131:43359)
0926 02:17:09.128581 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:09.128587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:09.128607 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:17:09.128779 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:09.128783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:09.128848 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:09.128956 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:09.130396 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:09.131150 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:09.132500 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:09.132896 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:09.133005 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:09.137623 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:09.137642 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:09.137660 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:09.137700 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:09.140328 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:09.141113 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:09.141122 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:09.141150 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:09.141332 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:09.141744 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:09.141750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:09.141754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:09.142202 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:09.142497 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:09.142908 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 02:17:09.143171 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:17:09.144249 12837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:17:09.234202 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 12713
I20260926 02:17:09.248073 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 12346
I20260926 02:17:09.249212 12410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:17:09.358733 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 12346
I20260926 02:17:29.376585 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34341
--webserver_interface=127.28.152.190
--webserver_port=40481
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:17:29.710878 12892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:29.711211 12892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:29.711334 12892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:29.718878 12892 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:17:29.718993 12892 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:17:29.719034 12892 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:17:29.719105 12892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:17:29.719178 12892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:29.719266 12892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:29.719324 12892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:17:29.719455 12892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:17:29.729971 12892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=40481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:29.731364 12892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:29.732849 12892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:29.755714 12899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:29.755770 12898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:29.755776 12901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:29.756629 12892 server_base.cc:1034] running on GCE node
I20260926 02:17:29.757679 12892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:29.759405 12892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:29.760684 12892 hybrid_clock.cc:648] HybridClock initialized: now 1790389049760590 us; error 70 us; skew 500 ppm
I20260926 02:17:29.761106 12892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:29 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389049, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:29.769040 12892 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:17:29.771071 12892 webserver.cc:460] Webserver started at http://127.28.152.190:40481/ using document root <none> and password file <none>
I20260926 02:17:29.771955 12892 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:29.772086 12892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:29.778957 12892 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:17:29.782568 12908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:29.784411 12892 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:17:29.784642 12892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c"
format_stamp: "Formatted at 2026-09-26 02:17:06 on dist-test-slave-hllj"
server_key: "cbb0996bae36f2dd3a3f2b7c8b6bd57a"
server_key_iv: "0c388916738c9f5257cc87b20a4b299b"
server_key_version: "encryptionkey@0"
I20260926 02:17:29.785928 12892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:29.800168 12892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:29.804841 12892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:29.805207 12892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:29.824520 12892 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:34341
I20260926 02:17:29.824542 12959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:34341 every 8 connection(s)
I20260926 02:17:29.826206 12892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:17:29.830822 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 12892
I20260926 02:17:29.831849 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:42041
--local_ip_for_outbound_sockets=127.28.152.129
--tserver_master_addrs=127.28.152.190:34341
--webserver_port=38991
--webserver_interface=127.28.152.129
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:17:29.835197 12961 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:17:29.839907 12961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Bootstrap starting.
I20260926 02:17:29.860368 12961 log.cc:824] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:17:29.873870 12961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:17:29.874411 12961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Bootstrap complete.
I20260926 02:17:29.883652 12961 raft_consensus.cc:348] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:29.884569 12961 raft_consensus.cc:729] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32e7fca9e3b4bb9ae0150cda876fe6c, State: Initialized, Role: FOLLOWER
I20260926 02:17:29.885269 12961 consensus_queue.cc:260] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:29.885515 12961 raft_consensus.cc:388] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:17:29.885636 12961 raft_consensus.cc:482] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:17:29.885814 12961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:17:29.890055 12961 raft_consensus.cc:504] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:29.890692 12961 leader_election.cc:302] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f32e7fca9e3b4bb9ae0150cda876fe6c; no voters:
I20260926 02:17:29.891368 12961 leader_election.cc:288] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:17:29.891521 12965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:17:29.893837 12961 sys_catalog.cc:564] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:17:29.897739 12965 raft_consensus.cc:686] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 2 LEADER]: Becoming Leader. State: Replica: f32e7fca9e3b4bb9ae0150cda876fe6c, State: Running, Role: LEADER
I20260926 02:17:29.898450 12965 consensus_queue.cc:237] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:29.901167 12966 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } } }
I20260926 02:17:29.901643 12966 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: This master's current role is: LEADER
I20260926 02:17:29.902192 12967 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f32e7fca9e3b4bb9ae0150cda876fe6c. Latest consensus state: current_term: 2 leader_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } } }
I20260926 02:17:29.902474 12967 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: This master's current role is: LEADER
I20260926 02:17:29.904557 12977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:17:29.917832 12977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:17:29.919682 12977 catalog_manager.cc:1259] Loaded cluster ID: d23bd841c6004e3582609bef4bec1293
I20260926 02:17:29.919813 12977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:17:29.925712 12977 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:17:29.927626 12977 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Loaded TSK: 0
I20260926 02:17:29.961391 12977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Generated new TSK 1
I20260926 02:17:29.961871 12977 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:17:30.195755 12963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:30.196235 12963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:30.196403 12963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:30.208181 12963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:30.208370 12963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:30.208575 12963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:17:30.224727 12963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:42041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=38991
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:30.226538 12963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:30.228485 12963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:30.244488 12990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:30.244488 12991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:30.245199 12993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:30.245647 12963 server_base.cc:1034] running on GCE node
I20260926 02:17:30.246277 12963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:30.247296 12963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:30.248558 12963 hybrid_clock.cc:648] HybridClock initialized: now 1790389050248487 us; error 64 us; skew 500 ppm
I20260926 02:17:30.248873 12963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:30 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:30.252341 12963 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:17:30.254156 12963 webserver.cc:460] Webserver started at http://127.28.152.129:38991/ using document root <none> and password file <none>
I20260926 02:17:30.255077 12963 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:30.255234 12963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:30.261252 12963 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 02:17:30.264951 13000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:30.266861 12963 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 02:17:30.267094 12963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "2a6b55930f1f49e1b1f120231d0bf80e"
format_stamp: "Formatted at 2026-09-26 02:17:07 on dist-test-slave-hllj"
server_key: "fdd509cce9fb292586503671becbd31a"
server_key_iv: "4027060e2f02faf573620d3eeb2e31ff"
server_key_version: "encryptionkey@0"
I20260926 02:17:30.268556 12963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:30.300216 12963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:30.304816 12963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:30.305148 12963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:30.306232 12963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:17:30.307655 12963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:17:30.307757 12963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:30.307842 12963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:17:30.307896 12963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:30.340502 12963 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:42041
I20260926 02:17:30.340600 13112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:42041 every 8 connection(s)
I20260926 02:17:30.341997 12963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:17:30.352255 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 12963
I20260926 02:17:30.353014 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:33409
--local_ip_for_outbound_sockets=127.28.152.130
--tserver_master_addrs=127.28.152.190:34341
--webserver_port=44909
--webserver_interface=127.28.152.130
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:17:30 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:17:30.371505 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:30.345289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:59221)
0926 02:17:30.346148 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:30.346189 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:30.347913 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:30.349624 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:30.349636 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:30.350234 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:17:30.350987 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:30.351009 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:30.353546 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:30.353553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:30.355926 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:30.355955 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:30.357609 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:30.357620 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:30.358023 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:30.359030 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:17:30.359055 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:17:30.361876 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:17:30.366793 (+ 4917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:30.366803 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:30.366822 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:30.367453 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:30.368032 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:30.368036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:30.368039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:30.368272 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:30.369227 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:17:30.369236 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:17:30.369689 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 02:17:30.370022 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 02:17:30.370473 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":335,"threads_started":1}
I20260926 02:17:30.371882 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:30.345097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:59221 (local address 127.28.152.190:34341)
0926 02:17:30.345815 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:30.345833 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:30.346335 (+ 502us) server_negotiation.cc:407] Connection header received
0926 02:17:30.348438 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:30.348452 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:30.348925 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:30.349413 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:30.351186 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:30.353305 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:30.356153 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:30.357259 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:30.357475 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:30.362459 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:30.362472 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:30.362499 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:30.362562 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:30.366561 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:30.367655 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:30.367665 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:30.367673 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:30.367894 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:30.368391 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:30.368396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:30.368398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:30.369024 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:30.369252 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:30.370275 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0926 02:17:30.370768 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":255,"threads_started":1}
I20260926 02:17:30.374830 13113 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34341
I20260926 02:17:30.375174 13113 heartbeater.cc:464] Registering TS with master...
I20260926 02:17:30.375946 13113 heartbeater.cc:511] Master 127.28.152.190:34341 requested a full tablet report, sending...
I20260926 02:17:30.378448 12925 ts_manager.cc:194] Registered new tserver with Master: 2a6b55930f1f49e1b1f120231d0bf80e (127.28.152.129:42041)
I20260926 02:17:30.380537 12925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:59221
W20260926 02:17:30.702183 13117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:30.702509 13117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:30.702636 13117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:30.710516 13117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:30.710678 13117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:30.710842 13117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:17:30.720748 13117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:33409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=44909
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:30.722179 13117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:30.723510 13117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:30.736747 13122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:30.736778 13123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:30.736812 13125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:30.737550 13117 server_base.cc:1034] running on GCE node
I20260926 02:17:30.738171 13117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:30.739133 13117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:30.740340 13117 hybrid_clock.cc:648] HybridClock initialized: now 1790389050740292 us; error 40 us; skew 500 ppm
I20260926 02:17:30.740660 13117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:30 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:30.743891 13117 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:17:30.745530 13117 webserver.cc:460] Webserver started at http://127.28.152.130:44909/ using document root <none> and password file <none>
I20260926 02:17:30.746232 13117 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:30.746369 13117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:30.752116 13117 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:17:30.755268 13132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:30.756774 13117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:17:30.756959 13117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "0d9a4ee5082042d58641ed5ec8b980f3"
format_stamp: "Formatted at 2026-09-26 02:17:07 on dist-test-slave-hllj"
server_key: "32843680e736cd7cdac60a2070a3257b"
server_key_iv: "b3489a57f4bd7fde63fb5b846ea3137f"
server_key_version: "encryptionkey@0"
I20260926 02:17:30.758033 13117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:30.786317 13117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:30.790911 13117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:30.791172 13117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:30.792212 13117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:17:30.793627 13117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:17:30.793720 13117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:30.793824 13117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:17:30.793893 13117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:30.825356 13117 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:33409
I20260926 02:17:30.825404 13246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:33409 every 8 connection(s)
I20260926 02:17:30.827190 13117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:17:30.834156 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 13117
I20260926 02:17:30.834925 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:43359
--local_ip_for_outbound_sockets=127.28.152.131
--tserver_master_addrs=127.28.152.190:34341
--webserver_port=42659
--webserver_interface=127.28.152.131
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:17:30 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:17:30.848477 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:30.830165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:51537 (local address 127.28.152.190:34341)
0926 02:17:30.830340 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:30.830344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:30.831814 (+ 1470us) server_negotiation.cc:407] Connection header received
0926 02:17:30.833747 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:30.833753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:30.833861 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:30.833947 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:30.835652 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:30.836369 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:30.838100 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:30.838564 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:30.838690 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:30.842447 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:30.842455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:30.842458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:30.842494 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:30.845357 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:30.846265 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:30.846268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:30.846271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:30.846396 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:30.846851 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:30.846854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:30.846856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:30.847128 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:30.847235 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:30.848058 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 02:17:30.848238 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:17:30.849993 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:30.830783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:51537)
0926 02:17:30.831632 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:30.831678 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:30.833456 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:30.834205 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:30.834214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:30.834791 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:17:30.835479 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:30.835499 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:30.836567 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:30.836572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:30.837943 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:30.837958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:30.838807 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:30.838814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:30.839052 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:30.839918 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:17:30.839942 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:17:30.842187 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:17:30.845528 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:30.845538 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:30.845560 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:30.846151 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:30.846491 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:30.846496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:30.846500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:30.846753 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:30.847247 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:17:30.847255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:17:30.847705 (+ 450us) client_negotiation.cc:769] Sending connection context
0926 02:17:30.847990 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 02:17:30.848516 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":332,"threads_started":1}
I20260926 02:17:30.851513 13247 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34341
I20260926 02:17:30.851830 13247 heartbeater.cc:464] Registering TS with master...
I20260926 02:17:30.852536 13247 heartbeater.cc:511] Master 127.28.152.190:34341 requested a full tablet report, sending...
I20260926 02:17:30.854059 12925 ts_manager.cc:194] Registered new tserver with Master: 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409)
I20260926 02:17:30.854984 12925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:51537
W20260926 02:17:31.170228 13250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:31.170594 13250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:31.170725 13250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:31.178156 13250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:31.178282 13250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:31.178428 13250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:17:31.188211 13250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:43359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=42659
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:31.189755 13250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:31.191303 13250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:31.204821 13255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:31.204816 13258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:31.204833 13256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:31.205886 13250 server_base.cc:1034] running on GCE node
I20260926 02:17:31.206521 13250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:31.207484 13250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:31.208679 13250 hybrid_clock.cc:648] HybridClock initialized: now 1790389051208625 us; error 44 us; skew 500 ppm
I20260926 02:17:31.208987 13250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:31 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389051, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:31.212152 13250 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:17:31.213707 13250 webserver.cc:460] Webserver started at http://127.28.152.131:42659/ using document root <none> and password file <none>
I20260926 02:17:31.214409 13250 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:31.214529 13250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:31.220284 13250 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:17:31.223286 13266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:31.224838 13250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:17:31.225013 13250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "4c46720e86fd46c988754878e59a9d61"
format_stamp: "Formatted at 2026-09-26 02:17:08 on dist-test-slave-hllj"
server_key: "08622747c0fed1b7dca61b42d8c233b0"
server_key_iv: "d6616eb1c0c23131bc1a50b9deeddf7e"
server_key_version: "encryptionkey@0"
I20260926 02:17:31.226082 13250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:31.257431 13250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:31.262144 13250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:31.262396 13250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:31.263455 13250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:17:31.264818 13250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:17:31.264910 13250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:31.265004 13250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:17:31.265079 13250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:31.296301 13250 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:43359
I20260926 02:17:31.296370 13379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:43359 every 8 connection(s)
I20260926 02:17:31.297860 13250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:17:31.303680 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 13250
Sep 26 02:17:31 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389051, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:17:31.320339 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.304644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.28.152.190:34341)
0926 02:17:31.305354 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.305358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.305391 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:17:31.305438 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.305442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.305495 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.305575 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:31.307338 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.308076 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.309792 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.310264 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.310354 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.313590 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:31.313597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:31.313601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:31.313654 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:31.316899 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.317874 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.317879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.317882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.318013 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.318504 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.318510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.318514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.319092 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:31.319342 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.319765 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.319954 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":620,"spinlock_wait_cycles":5248,"thread_start_us":520,"threads_started":1}
I20260926 02:17:31.320899 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.300327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:49161 (local address 127.28.152.190:34341)
0926 02:17:31.300506 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.300510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.301729 (+ 1219us) server_negotiation.cc:407] Connection header received
0926 02:17:31.303614 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.303619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.303700 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.303787 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:31.305751 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.306445 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.308306 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.308750 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.308840 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.313002 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:31.313007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:31.313010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:31.313045 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:31.316590 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.317707 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.317712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.317714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.317852 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.318504 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.318510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.318514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.319092 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:31.319255 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.320254 (+ 999us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.320615 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":16768}
I20260926 02:17:31.322108 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.300835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:49161)
0926 02:17:31.301553 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.301597 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:31.303322 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:31.304062 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:31.304071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:31.304665 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:17:31.305571 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.305601 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.306605 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.306611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.308163 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.308180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.308977 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.308984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.309308 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.310068 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:17:31.310092 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:17:31.312762 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:17:31.316799 (+ 4037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:31.316807 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:31.316823 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:31.317548 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:31.318004 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:31.318008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:31.318011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:31.318303 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:31.319260 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:17:31.319282 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:17:31.319804 (+ 522us) client_negotiation.cc:769] Sending connection context
0926 02:17:31.320145 (+ 341us) client_negotiation.cc:240] Negotiation successful
0926 02:17:31.320741 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":291,"threads_started":1}
I20260926 02:17:31.323669 13380 heartbeater.cc:347] Connected to a master server at 127.28.152.190:34341
I20260926 02:17:31.323979 13380 heartbeater.cc:464] Registering TS with master...
I20260926 02:17:31.324734 13380 heartbeater.cc:511] Master 127.28.152.190:34341 requested a full tablet report, sending...
I20260926 02:17:31.326231 12925 ts_manager.cc:194] Registered new tserver with Master: 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359)
I20260926 02:17:31.327041 12925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:49161
I20260926 02:17:31.336015 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:17:31.343497 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.336990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.28.152.190:34341)
0926 02:17:31.337221 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.337226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.337639 (+ 413us) server_negotiation.cc:407] Connection header received
0926 02:17:31.337861 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.337866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.338025 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.338170 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:17:31.338670 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.339444 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.341255 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.341932 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.342073 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.342591 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:17:31.342841 (+ 250us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 02:17:31.343235 (+ 394us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45608: token expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 02:17:31.343633 13115 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45608: token expired
W20260926 02:17:31.344136 12848 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.152.190:34341)
I20260926 02:17:31.357219 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.344593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.28.152.190:34341)
0926 02:17:31.344762 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.344767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.344857 (+ 90us) server_negotiation.cc:407] Connection header received
0926 02:17:31.344975 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.344980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.345051 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.345171 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:31.346530 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.347262 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.348555 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.348939 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.349034 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.351787 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:31.351792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:31.351795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:31.351830 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:31.354654 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.355352 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.355356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.355358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.355475 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.355856 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.355859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.355861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.356223 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:31.356427 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.356751 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.356928 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:17:31.360821 12925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:17:31.368286 12925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:17:31.384636 13113 heartbeater.cc:503] Master 127.28.152.190:34341 was elected leader, sending a full tablet report...
I20260926 02:17:31.403637 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.380854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.28.152.129:42041)
0926 02:17:31.392126 (+ 11272us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.392139 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.395836 (+ 3697us) server_negotiation.cc:407] Connection header received
0926 02:17:31.395928 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.395935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.396191 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.396369 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:17:31.396972 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.398589 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.401479 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.402465 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.402669 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.402877 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.403116 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.403284 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11021,"thread_start_us":11297,"threads_started":1}
I20260926 02:17:31.403759 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.380744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42041 (local address 127.0.0.1:55314)
0926 02:17:31.394111 (+ 13367us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.394126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:31.394257 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:31.396370 (+ 2113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:31.396375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:31.396399 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:17:31.396731 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.396743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.398746 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.398751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.401274 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.401292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.402680 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.402686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.402879 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.402931 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:17:31.403064 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 02:17:31.403264 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13222,"mutex_wait_us":43,"thread_start_us":11820,"threads_started":1}
I20260926 02:17:31.404474 13394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.394304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33409 (local address 127.0.0.1:48978)
0926 02:17:31.395001 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.395019 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:31.395145 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:31.397598 (+ 2453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:31.397602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:31.397627 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:17:31.398050 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.398064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.399708 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.399714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.401371 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.401387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.403490 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.403498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.403733 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.403829 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 02:17:31.403913 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 02:17:31.404087 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":47,"thread_start_us":357,"threads_started":1}
I20260926 02:17:31.404534 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.380624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.28.152.131:43359)
0926 02:17:31.394032 (+ 13408us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.394051 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.394086 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:17:31.394192 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.394200 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.394486 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.394706 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:17:31.395713 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.397337 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.401447 (+ 4110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.402962 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.403144 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.403291 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.404030 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.404202 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13241,"thread_start_us":13069,"threads_started":1}
I20260926 02:17:31.405056 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.380337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:43359 (local address 127.0.0.1:45990)
0926 02:17:31.393056 (+ 12719us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.393120 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:31.393359 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:31.395012 (+ 1653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:31.395020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:31.395063 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:17:31.395502 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.395516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.397468 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.397473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.401287 (+ 3814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.401296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.403525 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.403531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.403734 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.403831 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 02:17:31.403913 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 02:17:31.404748 (+ 835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12398,"mutex_wait_us":765,"spinlock_wait_cycles":1280,"thread_start_us":13528,"threads_started":1}
I20260926 02:17:31.405764 13395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.394566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.28.152.130:33409)
0926 02:17:31.397002 (+ 2436us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.397011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.397037 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:17:31.397159 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.397166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.397415 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.397615 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:17:31.398249 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.399590 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.401555 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.402958 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.403123 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.403291 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.405282 (+ 1991us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.405461 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2268,"thread_start_us":2125,"threads_started":1}
I20260926 02:17:31.408576 13181 tablet_service.cc:1467] Processing CreateTablet for tablet 3c19750a0f72407bb85ddb77e90e4f49 (DEFAULT_TABLE table=test-table [id=9183c54bfbf84f4082abb5a4732336ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:17:31.408608 13314 tablet_service.cc:1467] Processing CreateTablet for tablet 3c19750a0f72407bb85ddb77e90e4f49 (DEFAULT_TABLE table=test-table [id=9183c54bfbf84f4082abb5a4732336ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:17:31.408608 13048 tablet_service.cc:1467] Processing CreateTablet for tablet 3c19750a0f72407bb85ddb77e90e4f49 (DEFAULT_TABLE table=test-table [id=9183c54bfbf84f4082abb5a4732336ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:17:31.409843 13181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c19750a0f72407bb85ddb77e90e4f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:17:31.409981 13314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c19750a0f72407bb85ddb77e90e4f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:17:31.409994 13048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c19750a0f72407bb85ddb77e90e4f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:17:31.422976 13396 tablet_bootstrap.cc:492] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3: Bootstrap starting.
I20260926 02:17:31.425225 13397 tablet_bootstrap.cc:492] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: Bootstrap starting.
I20260926 02:17:31.426016 13398 tablet_bootstrap.cc:492] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61: Bootstrap starting.
I20260926 02:17:31.427203 13396 tablet_bootstrap.cc:654] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3: Neither blocks nor log segments found. Creating new log.
I20260926 02:17:31.428674 13396 log.cc:824] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:17:31.429633 13398 tablet_bootstrap.cc:654] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61: Neither blocks nor log segments found. Creating new log.
I20260926 02:17:31.430723 13398 log.cc:824] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61: Log is configured to *not* fsync() on all Append() calls
I20260926 02:17:31.432247 13396 tablet_bootstrap.cc:492] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3: No bootstrap required, opened a new log
I20260926 02:17:31.432452 13397 tablet_bootstrap.cc:654] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: Neither blocks nor log segments found. Creating new log.
I20260926 02:17:31.432600 13396 ts_tablet_manager.cc:1397] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 02:17:31.433799 13397 log.cc:824] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:17:31.435334 13398 tablet_bootstrap.cc:492] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61: No bootstrap required, opened a new log
I20260926 02:17:31.435595 13398 ts_tablet_manager.cc:1397] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 02:17:31.436944 13397 tablet_bootstrap.cc:492] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: No bootstrap required, opened a new log
I20260926 02:17:31.437245 13397 ts_tablet_manager.cc:1397] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 02:17:31.442776 13398 raft_consensus.cc:348] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:31.442842 13396 raft_consensus.cc:348] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:31.443171 13398 raft_consensus.cc:374] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:17:31.443267 13398 raft_consensus.cc:729] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c46720e86fd46c988754878e59a9d61, State: Initialized, Role: FOLLOWER
I20260926 02:17:31.443336 13396 raft_consensus.cc:374] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:17:31.443468 13396 raft_consensus.cc:729] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9a4ee5082042d58641ed5ec8b980f3, State: Initialized, Role: FOLLOWER
I20260926 02:17:31.443933 13398 consensus_queue.cc:260] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:31.444118 13396 consensus_queue.cc:260] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:31.445747 13398 ts_tablet_manager.cc:1428] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260926 02:17:31.446270 13380 heartbeater.cc:503] Master 127.28.152.190:34341 was elected leader, sending a full tablet report...
I20260926 02:17:31.447976 13397 raft_consensus.cc:348] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:31.448381 13247 heartbeater.cc:503] Master 127.28.152.190:34341 was elected leader, sending a full tablet report...
I20260926 02:17:31.448498 13397 raft_consensus.cc:374] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:17:31.448624 13397 raft_consensus.cc:729] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6b55930f1f49e1b1f120231d0bf80e, State: Initialized, Role: FOLLOWER
I20260926 02:17:31.449069 13396 ts_tablet_manager.cc:1428] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
I20260926 02:17:31.449337 13397 consensus_queue.cc:260] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:31.454449 13397 ts_tablet_manager.cc:1428] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260926 02:17:31.470566 13402 raft_consensus.cc:482] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:17:31.470870 13402 raft_consensus.cc:504] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:31.472568 13402 leader_election.cc:288] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a6b55930f1f49e1b1f120231d0bf80e (127.28.152.129:42041), 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409)
I20260926 02:17:31.479038 13395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.473040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:33823 (local address 127.28.152.130:33409)
0926 02:17:31.473272 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.473278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.473428 (+ 150us) server_negotiation.cc:407] Connection header received
0926 02:17:31.473609 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.473615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.473701 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.473808 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:17:31.474289 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.475034 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.476996 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.478220 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.478357 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.478412 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.478746 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.478840 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 02:17:31.479144 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.473114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33409 (local address 127.28.152.131:33823)
0926 02:17:31.473318 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.473342 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:31.473471 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:31.473850 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:31.473859 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:31.473887 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:17:31.474110 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.474123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.475156 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.475161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.476826 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.476838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.478372 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.478377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.478586 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.478622 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:17:31.478730 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 02:17:31.478864 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 02:17:31.480042 13202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c19750a0f72407bb85ddb77e90e4f49" candidate_uuid: "4c46720e86fd46c988754878e59a9d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" is_pre_election: true
I20260926 02:17:31.480428 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.473107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:50891 (local address 127.28.152.129:42041)
0926 02:17:31.473330 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.473335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.474221 (+ 886us) server_negotiation.cc:407] Connection header received
0926 02:17:31.474376 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.474380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.474441 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.474515 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:17:31.475045 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.476136 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.478336 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.479568 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.479709 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.479779 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.480060 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.480192 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 02:17:31.480490 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.473167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42041 (local address 127.28.152.131:50891)
0926 02:17:31.474110 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.474128 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:31.474242 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:31.474540 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:31.474544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:31.474566 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:17:31.474868 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.474877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.476308 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.476314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.478144 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:31.478160 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.479739 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.479745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:31.479895 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.479918 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:17:31.480018 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:17:31.480138 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":136,"thread_start_us":352,"threads_started":1}
I20260926 02:17:31.480530 13202 raft_consensus.cc:2454] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c46720e86fd46c988754878e59a9d61 in term 0.
I20260926 02:17:31.481335 13267 leader_election.cc:302] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9a4ee5082042d58641ed5ec8b980f3, 4c46720e86fd46c988754878e59a9d61; no voters:
I20260926 02:17:31.481757 13402 raft_consensus.cc:2781] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:17:31.481663 13068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c19750a0f72407bb85ddb77e90e4f49" candidate_uuid: "4c46720e86fd46c988754878e59a9d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" is_pre_election: true
I20260926 02:17:31.481916 13402 raft_consensus.cc:482] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:17:31.482016 13402 raft_consensus.cc:3037] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:17:31.482249 13068 raft_consensus.cc:2454] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c46720e86fd46c988754878e59a9d61 in term 0.
I20260926 02:17:31.484498 13402 raft_consensus.cc:504] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:31.485080 13402 leader_election.cc:288] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [CANDIDATE]: Term 1 election: Requested vote from peers 2a6b55930f1f49e1b1f120231d0bf80e (127.28.152.129:42041), 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409)
I20260926 02:17:31.485522 13068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c19750a0f72407bb85ddb77e90e4f49" candidate_uuid: "4c46720e86fd46c988754878e59a9d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6b55930f1f49e1b1f120231d0bf80e"
I20260926 02:17:31.485615 13202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c19750a0f72407bb85ddb77e90e4f49" candidate_uuid: "4c46720e86fd46c988754878e59a9d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9a4ee5082042d58641ed5ec8b980f3"
I20260926 02:17:31.485702 13068 raft_consensus.cc:3037] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:17:31.485821 13202 raft_consensus.cc:3037] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:17:31.487910 13068 raft_consensus.cc:2454] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c46720e86fd46c988754878e59a9d61 in term 1.
I20260926 02:17:31.487995 13202 raft_consensus.cc:2454] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c46720e86fd46c988754878e59a9d61 in term 1.
I20260926 02:17:31.488420 13267 leader_election.cc:302] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9a4ee5082042d58641ed5ec8b980f3, 4c46720e86fd46c988754878e59a9d61; no voters:
I20260926 02:17:31.488783 13402 raft_consensus.cc:2781] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:17:31.489239 13402 raft_consensus.cc:686] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 1 LEADER]: Becoming Leader. State: Replica: 4c46720e86fd46c988754878e59a9d61, State: Running, Role: LEADER
I20260926 02:17:31.489703 13402 consensus_queue.cc:237] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:31.494554 12925 catalog_manager.cc:5495] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c46720e86fd46c988754878e59a9d61 (127.28.152.131). New cstate: current_term: 1 leader_uuid: "4c46720e86fd46c988754878e59a9d61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } health_report { overall_health: UNKNOWN } } }
I20260926 02:17:31.517927 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.512668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.28.152.131:43359)
0926 02:17:31.512868 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.512872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.512890 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:17:31.513097 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.513101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.513216 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.513345 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:17:31.513726 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.514475 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.515852 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.516262 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.516358 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.516820 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:17:31.517373 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:17:31.517491 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.517599 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.517670 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:17:31.526680 13202 raft_consensus.cc:1261] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3 [term 1 FOLLOWER]: Refusing update from remote peer 4c46720e86fd46c988754878e59a9d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:17:31.526645 13068 raft_consensus.cc:1261] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Refusing update from remote peer 4c46720e86fd46c988754878e59a9d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:17:31.527788 13402 consensus_queue.cc:1035] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:17:31.528112 13406 consensus_queue.cc:1035] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:17:31.558794 12924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:17:31.559085 12924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:17:31.562848 12924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Sending DeleteTablet for 3 replicas of tablet 3c19750a0f72407bb85ddb77e90e4f49
I20260926 02:17:31.564170 13314 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c19750a0f72407bb85ddb77e90e4f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:17:31 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:45990
I20260926 02:17:31.564191 13048 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c19750a0f72407bb85ddb77e90e4f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:17:31 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:55314
I20260926 02:17:31.565084 13419 tablet_replica.cc:330] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61: stopping tablet replica
I20260926 02:17:31.565188 13181 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c19750a0f72407bb85ddb77e90e4f49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:17:31 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:48978
I20260926 02:17:31.565629 13419 raft_consensus.cc:2229] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:17:31.566207 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 12963
I20260926 02:17:31.566445 13419 raft_consensus.cc:2258] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:17:31.568897 13419 ts_tablet_manager.cc:1905] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:17:31.577255 13419 ts_tablet_manager.cc:1918] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:17:31.577695 13419 log.cc:1196] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/3c19750a0f72407bb85ddb77e90e4f49
I20260926 02:17:31.578401 13419 ts_tablet_manager.cc:1939] T 3c19750a0f72407bb85ddb77e90e4f49 P 4c46720e86fd46c988754878e59a9d61: Deleting consensus metadata
I20260926 02:17:31.580580 12909 catalog_manager.cc:4841] TS 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359): tablet 3c19750a0f72407bb85ddb77e90e4f49 (table test-table [id=9183c54bfbf84f4082abb5a4732336ad]) successfully deleted
I20260926 02:17:31.581657 13421 tablet_replica.cc:330] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3: stopping tablet replica
I20260926 02:17:31.582048 13421 raft_consensus.cc:2229] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:17:31.582495 13421 raft_consensus.cc:2258] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:17:31.584462 13421 ts_tablet_manager.cc:1905] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:17:31.585449 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.567389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.28.152.129:42041)
0926 02:17:31.567557 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.567562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.567582 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:17:31.567642 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.567647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.567734 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.567838 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:31.569573 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.570783 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.572864 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.573566 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.573709 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.577404 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:31.577424 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:31.577447 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:31.577517 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:31.581668 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.582795 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.582807 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.582810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.582987 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.583458 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.583463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.583467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.584089 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:31.584425 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.584777 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.585042 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:17:31.586190 13108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:17:31.592988 13421 ts_tablet_manager.cc:1918] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:17:31.593370 13421 log.cc:1196] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/3c19750a0f72407bb85ddb77e90e4f49
I20260926 02:17:31.594072 13421 ts_tablet_manager.cc:1939] T 3c19750a0f72407bb85ddb77e90e4f49 P 0d9a4ee5082042d58641ed5ec8b980f3: Deleting consensus metadata
I20260926 02:17:31.596472 12909 catalog_manager.cc:4841] TS 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409): tablet 3c19750a0f72407bb85ddb77e90e4f49 (table test-table [id=9183c54bfbf84f4082abb5a4732336ad]) successfully deleted
I20260926 02:17:31.688079 13420 tablet_replica.cc:330] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: stopping tablet replica
I20260926 02:17:31.688544 13420 raft_consensus.cc:2229] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:17:31.689014 13420 raft_consensus.cc:2258] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:17:31.690106 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 12963
I20260926 02:17:31.705241 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 13117
W20260926 02:17:31.705284 12910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.152.129:42041 (error 108)
I20260926 02:17:31.707626 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.707131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42041 (local address 127.0.0.1:55342)
0926 02:17:31.707316 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.707428 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 02:17:31.707909 12910 catalog_manager.cc:4584] TS 2a6b55930f1f49e1b1f120231d0bf80e (127.28.152.129:42041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c19750a0f72407bb85ddb77e90e4f49: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42041: connect: Connection refused (error 111)
I20260926 02:17:31.721659 13395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.706024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.28.152.130:33409)
0926 02:17:31.706205 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.706210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.706520 (+ 310us) server_negotiation.cc:407] Connection header received
0926 02:17:31.706596 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.706602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.706699 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.706805 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:31.708458 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.709682 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.711238 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.711874 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.712005 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.714739 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:31.714760 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:31.714783 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:31.714845 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:31.718671 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.719423 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.719433 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.719437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.719598 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.719926 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.719931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.719934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.720433 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:31.720735 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.721024 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.721285 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 02:17:31.722390 13242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:17:31.761703 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.761094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42041 (local address 127.0.0.1:55352)
0926 02:17:31.761354 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.761491 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 02:17:31.763053 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.762610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42041 (local address 127.0.0.1:55364)
0926 02:17:31.762780 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.762890 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 02:17:31.812351 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 13117
I20260926 02:17:31.824051 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.823477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42041 (local address 127.0.0.1:55380)
0926 02:17:31.823728 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.823846 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 02:17:31.825503 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.825085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42041 (local address 127.0.0.1:55388)
0926 02:17:31.825252 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.825345 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 02:17:31.827582 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 13250
I20260926 02:17:31.841969 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.828300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.28.152.131:43359)
0926 02:17:31.828490 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:31.828496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:31.828519 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:17:31.828701 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:31.828707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:31.828771 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:31.828860 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:31.830462 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.831341 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.832725 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:31.833230 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:31.833358 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:31.836160 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:31.836174 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:31.836190 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:31.836226 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:31.839080 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.839821 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.839828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.839830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.839989 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:31.840365 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:31.840368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:31.840370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:31.840745 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:31.841063 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:31.841365 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 02:17:31.841591 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:17:31.842721 13375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:17:31.906723 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.906162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42041 (local address 127.0.0.1:55400)
0926 02:17:31.906412 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.906544 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 02:17:31.908011 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:31.907543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42041 (local address 127.0.0.1:55410)
0926 02:17:31.907767 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:31.907861 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 02:17:31.964938 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 13250
I20260926 02:17:31.981547 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 12892
I20260926 02:17:31.982712 12955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:17:32.095458 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:32.094851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42041 (local address 127.0.0.1:55418)
0926 02:17:32.095056 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:32.095174 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 02:17:32.095710 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 12892
I20260926 02:17:52.114184 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34341
--webserver_interface=127.28.152.190
--webserver_port=40481
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:17:52.451720 13427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:52.452037 13427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:52.452143 13427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:52.459573 13427 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:17:52.459693 13427 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:17:52.459730 13427 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:17:52.459798 13427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:17:52.459874 13427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:52.459954 13427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:52.459992 13427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:17:52.460038 13427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:17:52.470325 13427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=40481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:52.471755 13427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:52.473122 13427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:52.485455 13433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:52.485487 13432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:52.485386 13435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:52.486213 13427 server_base.cc:1034] running on GCE node
I20260926 02:17:52.487111 13427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:52.488615 13427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:52.489837 13427 hybrid_clock.cc:648] HybridClock initialized: now 1790389072489768 us; error 49 us; skew 500 ppm
I20260926 02:17:52.490170 13427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:52 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:52.493433 13427 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:17:52.494976 13427 webserver.cc:460] Webserver started at http://127.28.152.190:40481/ using document root <none> and password file <none>
I20260926 02:17:52.495724 13427 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:52.495831 13427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:52.502224 13427 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:17:52.505280 13442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:52.506773 13427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:17:52.506969 13427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c"
format_stamp: "Formatted at 2026-09-26 02:17:06 on dist-test-slave-hllj"
server_key: "cbb0996bae36f2dd3a3f2b7c8b6bd57a"
server_key_iv: "0c388916738c9f5257cc87b20a4b299b"
server_key_version: "encryptionkey@0"
I20260926 02:17:52.507948 13427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:52.545025 13427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:52.549844 13427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:52.550112 13427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:52.568504 13427 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:34341
I20260926 02:17:52.568527 13495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:34341 every 8 connection(s)
I20260926 02:17:52.570227 13427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:17:52.577831 13496 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:17:52.578656 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 13427
I20260926 02:17:52.579479 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:42041
--local_ip_for_outbound_sockets=127.28.152.129
--tserver_master_addrs=127.28.152.190:34341
--webserver_port=38991
--webserver_interface=127.28.152.129
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:17:52.581732 13496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Bootstrap starting.
I20260926 02:17:52.602646 13496 log.cc:824] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:17:52.620201 13496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:17:52.620839 13496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Bootstrap complete.
I20260926 02:17:52.630044 13496 raft_consensus.cc:348] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:52.631036 13496 raft_consensus.cc:729] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32e7fca9e3b4bb9ae0150cda876fe6c, State: Initialized, Role: FOLLOWER
I20260926 02:17:52.631754 13496 consensus_queue.cc:260] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:52.631995 13496 raft_consensus.cc:388] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:17:52.632104 13496 raft_consensus.cc:482] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:17:52.632275 13496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:17:52.636274 13496 raft_consensus.cc:504] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:52.636792 13496 leader_election.cc:302] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f32e7fca9e3b4bb9ae0150cda876fe6c; no voters:
I20260926 02:17:52.637458 13496 leader_election.cc:288] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 02:17:52.637609 13500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 3 FOLLOWER]: Leader election won for term 3
I20260926 02:17:52.638631 13500 raft_consensus.cc:686] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [term 3 LEADER]: Becoming Leader. State: Replica: f32e7fca9e3b4bb9ae0150cda876fe6c, State: Running, Role: LEADER
I20260926 02:17:52.639299 13500 consensus_queue.cc:237] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } }
I20260926 02:17:52.639632 13496 sys_catalog.cc:564] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:17:52.641923 13502 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f32e7fca9e3b4bb9ae0150cda876fe6c. Latest consensus state: current_term: 3 leader_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } } }
I20260926 02:17:52.642314 13502 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: This master's current role is: LEADER
I20260926 02:17:52.642184 13501 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32e7fca9e3b4bb9ae0150cda876fe6c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34341 } } }
I20260926 02:17:52.642525 13501 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c [sys.catalog]: This master's current role is: LEADER
I20260926 02:17:52.652180 13508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:17:52.660959 13508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:17:52.662871 13508 catalog_manager.cc:1259] Loaded cluster ID: d23bd841c6004e3582609bef4bec1293
I20260926 02:17:52.663095 13508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:17:52.667857 13508 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:17:52.670014 13508 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Loaded TSK: 0
I20260926 02:17:52.670272 13508 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f32e7fca9e3b4bb9ae0150cda876fe6c: Loaded TSK: 1
F20260926 02:17:52.671607 13508 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa74f81b9ad google::LogMessage::Fail() at ??:0
@ 0x7fa74f81e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa74f81b4de google::LogMessage::Flush() at ??:0
@ 0x7fa74f81ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa75357f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa75356e73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa76524a1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa765249783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa7652d52a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa7652d50c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa76531f5e9 std::function<>::operator()() at ??:0
@ 0x7fa76524dcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa76524bc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa7652d3610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa7652d3439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa752467345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa75246a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa75246a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa752445eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa75571f6db start_thread at ??:0
@ 0x7fa74e6b561f clone at ??:0
W20260926 02:17:52.981520 13498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:52.981820 13498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:52.981918 13498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:52.989351 13498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:52.989481 13498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:52.989603 13498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:17:52.999732 13498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:42041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=38991
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:53.001026 13498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:53.002398 13498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:53.015777 13544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:53.015967 13546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:53.015782 13543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:53.016629 13498 server_base.cc:1034] running on GCE node
I20260926 02:17:53.017346 13498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:53.018349 13498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:53.019586 13498 hybrid_clock.cc:648] HybridClock initialized: now 1790389073019532 us; error 47 us; skew 500 ppm
I20260926 02:17:53.019923 13498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:53 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:53.023418 13498 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:17:53.025050 13498 webserver.cc:460] Webserver started at http://127.28.152.129:38991/ using document root <none> and password file <none>
I20260926 02:17:53.025833 13498 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:53.025965 13498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:53.031996 13498 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:17:53.035598 13553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:53.037303 13498 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:17:53.037492 13498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "2a6b55930f1f49e1b1f120231d0bf80e"
format_stamp: "Formatted at 2026-09-26 02:17:07 on dist-test-slave-hllj"
server_key: "fdd509cce9fb292586503671becbd31a"
server_key_iv: "4027060e2f02faf573620d3eeb2e31ff"
server_key_version: "encryptionkey@0"
I20260926 02:17:53.038523 13498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:53.057943 13498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:53.062544 13498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:53.062829 13498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:53.063881 13498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:17:53.065863 13560 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 02:17:53.071920 13498 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 02:17:53.072081 13498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260926 02:17:53.072227 13498 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 02:17:53.075716 13498 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 02:17:53.075810 13498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.001s sys 0.000s
I20260926 02:17:53.076254 13560 tablet_bootstrap.cc:492] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: Bootstrap starting.
I20260926 02:17:53.085197 13560 log.cc:824] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:17:53.101590 13560 tablet_bootstrap.cc:492] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:17:53.103492 13560 tablet_bootstrap.cc:492] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: Bootstrap complete.
I20260926 02:17:53.104152 13560 ts_tablet_manager.cc:1397] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: Time spent bootstrapping tablet: real 0.028s user 0.025s sys 0.000s
I20260926 02:17:53.113866 13560 raft_consensus.cc:348] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:53.114881 13560 raft_consensus.cc:729] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6b55930f1f49e1b1f120231d0bf80e, State: Initialized, Role: FOLLOWER
I20260926 02:17:53.115584 13560 consensus_queue.cc:260] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:53.117298 13560 ts_tablet_manager.cc:1428] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260926 02:17:53.117848 13498 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:42041
I20260926 02:17:53.117867 13669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:42041 every 8 connection(s)
I20260926 02:17:53.119120 13498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:17:53.124017 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:53.122092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:39411)
0926 02:17:53.122859 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:53.123127 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":305,"threads_started":1}
I20260926 02:17:53.124480 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 13498
W20260926 02:17:53.124622 13670 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:34341 (0 consecutive failures): Network error: Failed to ping master at 127.28.152.190:34341: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
I20260926 02:17:53.125214 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:33409
--local_ip_for_outbound_sockets=127.28.152.130
--tserver_master_addrs=127.28.152.190:34341
--webserver_port=44909
--webserver_interface=127.28.152.130
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:17:53.125512 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:53.125117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:53347)
0926 02:17:53.125308 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:53.125386 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 02:17:53.126360 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:53.126020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:51893)
0926 02:17:53.126168 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:53.126243 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 02:17:53.126600 13670 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:17:53.455993 13673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:53.456315 13673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:53.456441 13673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:53.464002 13673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:53.464154 13673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:53.464303 13673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:17:53.474905 13673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:33409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=44909
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:53.476262 13673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:53.477666 13673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:53.491146 13679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:53.491194 13678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:53.491214 13681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:53.492216 13673 server_base.cc:1034] running on GCE node
I20260926 02:17:53.492842 13673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:53.493896 13673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:53.495146 13673 hybrid_clock.cc:648] HybridClock initialized: now 1790389073495072 us; error 57 us; skew 500 ppm
I20260926 02:17:53.495548 13673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:53 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:53.498920 13673 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:17:53.500504 13673 webserver.cc:460] Webserver started at http://127.28.152.130:44909/ using document root <none> and password file <none>
I20260926 02:17:53.501282 13673 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:53.501411 13673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:53.507238 13673 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:17:53.510203 13688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:53.511893 13673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:17:53.512066 13673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "0d9a4ee5082042d58641ed5ec8b980f3"
format_stamp: "Formatted at 2026-09-26 02:17:07 on dist-test-slave-hllj"
server_key: "32843680e736cd7cdac60a2070a3257b"
server_key_iv: "b3489a57f4bd7fde63fb5b846ea3137f"
server_key_version: "encryptionkey@0"
I20260926 02:17:53.513060 13673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:53.535039 13673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:53.539613 13673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:53.539897 13673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:53.540928 13673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:17:53.542459 13673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:17:53.542569 13673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:53.542689 13673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:17:53.542761 13673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:53.574501 13673 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:33409
I20260926 02:17:53.574550 13801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:33409 every 8 connection(s)
I20260926 02:17:53.576021 13673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:17:53.579238 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 13673
I20260926 02:17:53.579918 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:43359
--local_ip_for_outbound_sockets=127.28.152.131
--tserver_master_addrs=127.28.152.190:34341
--webserver_port=42659
--webserver_interface=127.28.152.131
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:17:53.581506 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:53.579200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:41607)
0926 02:17:53.580095 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:53.580424 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":355,"threads_started":1}
W20260926 02:17:53.582384 13802 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:34341 (0 consecutive failures): Network error: Failed to ping master at 127.28.152.190:34341: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
I20260926 02:17:53.583432 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:53.583017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:38215)
0926 02:17:53.583232 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:53.583311 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 02:17:53.584386 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:53.584028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:43655)
0926 02:17:53.584184 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:53.584264 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 02:17:53.584638 13802 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:17:53.912889 13805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:17:53.913237 13805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:17:53.913393 13805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:17:53.921013 13805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:17:53.921182 13805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:17:53.921363 13805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:17:53.931703 13805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:43359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=42659
--tserver_master_addrs=127.28.152.190:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:17:53.933085 13805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:17:53.934481 13805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:17:53.948032 13813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:53.948091 13811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:17:53.948280 13810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:17:53.949182 13805 server_base.cc:1034] running on GCE node
I20260926 02:17:53.949838 13805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:17:53.950841 13805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:17:53.952069 13805 hybrid_clock.cc:648] HybridClock initialized: now 1790389073951999 us; error 83 us; skew 500 ppm
I20260926 02:17:53.952420 13805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:17:53 dist-test-slave-hllj krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:17:53.955706 13805 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:17:53.957336 13805 webserver.cc:460] Webserver started at http://127.28.152.131:42659/ using document root <none> and password file <none>
I20260926 02:17:53.958068 13805 fs_manager.cc:359] Metadata directory not provided
I20260926 02:17:53.958196 13805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:17:53.964402 13805 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 02:17:53.967610 13820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:53.969257 13805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:17:53.969444 13805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "4c46720e86fd46c988754878e59a9d61"
format_stamp: "Formatted at 2026-09-26 02:17:08 on dist-test-slave-hllj"
server_key: "08622747c0fed1b7dca61b42d8c233b0"
server_key_iv: "d6616eb1c0c23131bc1a50b9deeddf7e"
server_key_version: "encryptionkey@0"
I20260926 02:17:53.970481 13805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:17:53.988960 13805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:17:53.993523 13805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:17:53.993788 13805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:17:53.994791 13805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:17:53.996176 13805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:17:53.996274 13805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:53.996387 13805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:17:53.996464 13805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:17:54.027493 13805 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:43359
I20260926 02:17:54.027530 13932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:43359 every 8 connection(s)
I20260926 02:17:54.029053 13805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:17:54.034377 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 13805
I20260926 02:17:54.034394 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:54.032260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:60343)
0926 02:17:54.033014 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:54.033307 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":301,"threads_started":1}
W20260926 02:17:54.035210 13933 heartbeater.cc:644] Failed to heartbeat to 127.28.152.190:34341 (0 consecutive failures): Network error: Failed to ping master at 127.28.152.190:34341: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
I20260926 02:17:54.036173 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:54.035769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:46077)
0926 02:17:54.035931 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:54.036012 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:17:54.037122 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:54.036765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:40025)
0926 02:17:54.036906 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:54.036995 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:17:54.037410 13933 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:17:54.128034 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:54.127205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:50399)
0926 02:17:54.127694 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:54.127783 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":250,"threads_started":1}
I20260926 02:17:54.494679 13938 raft_consensus.cc:482] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:17:54.494956 13938 raft_consensus.cc:504] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:54.496636 13938 leader_election.cc:288] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359), 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409)
Sep 26 02:17:54 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
Sep 26 02:17:54 dist-test-slave-hllj krb5kdc[12333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:17:54.519488 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:54.497307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:60007 (local address 127.28.152.131:43359)
0926 02:17:54.498802 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:54.498819 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:54.498871 (+ 52us) server_negotiation.cc:407] Connection header received
0926 02:17:54.500679 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:54.500692 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:54.501182 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:54.501477 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:54.503797 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:54.505039 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:54.506964 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:54.507646 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:54.507826 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:54.512054 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:54.512063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:54.512079 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:54.512132 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:54.515593 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:54.516452 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:54.516460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:54.516466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:54.516674 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:54.517185 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:54.517190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:54.517194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:54.517698 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:54.517855 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:54.518683 (+ 828us) server_negotiation.cc:299] Negotiation successful
0926 02:17:54.519063 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":1168,"threads_started":1}
I20260926 02:17:54.519496 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:54.497875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:35627 (local address 127.28.152.130:33409)
0926 02:17:54.499432 (+ 1557us) server_negotiation.cc:206] Beginning negotiation
0926 02:17:54.499443 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:17:54.499480 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:17:54.500977 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:17:54.500986 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:17:54.501438 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:17:54.501801 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:17:54.503792 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:54.505038 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:54.506946 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:17:54.507630 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:17:54.507808 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:54.512002 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:17:54.512011 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:17:54.512030 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:17:54.512077 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:17:54.515326 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:54.516334 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:54.516344 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:54.516352 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:54.516592 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:17:54.517086 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:17:54.517092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:17:54.517096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:17:54.517741 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:17:54.517943 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 02:17:54.518682 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 02:17:54.519049 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1395,"thread_start_us":1267,"threads_started":1}
I20260926 02:17:54.519518 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:54.497502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33409 (local address 127.28.152.129:35627)
0926 02:17:54.499137 (+ 1635us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:54.499156 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:54.499279 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:54.502186 (+ 2907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:54.502191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:54.502784 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:17:54.503565 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:54.503579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:54.505279 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:54.505284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:54.506782 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:54.506797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:54.507842 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:54.507849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:54.508154 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:54.508970 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:17:54.508987 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:17:54.511668 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:17:54.515498 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:54.515509 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:54.515528 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:54.516194 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:54.516723 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:54.516728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:54.516731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:54.516968 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:54.517890 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:17:54.517894 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:17:54.518162 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 02:17:54.518428 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 02:17:54.519063 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1499,"mutex_wait_us":275,"spinlock_wait_cycles":4096,"thread_start_us":1334,"threads_started":1}
I20260926 02:17:54.519591 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:54.497019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:43359 (local address 127.28.152.129:60007)
0926 02:17:54.497270 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:54.497317 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:17:54.497993 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:17:54.501709 (+ 3716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:17:54.501720 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:17:54.502610 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:17:54.503546 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:54.503569 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:54.505287 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:54.505291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:54.506781 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:17:54.506797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:17:54.507893 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:17:54.507901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:17:54.508149 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:17:54.508921 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:17:54.508951 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:17:54.511770 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:17:54.515748 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:54.515751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:54.515754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:54.516295 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:54.516794 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:17:54.516798 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:17:54.516801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:17:54.516974 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:17:54.517842 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:17:54.517851 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:17:54.518163 (+ 312us) client_negotiation.cc:769] Sending connection context
0926 02:17:54.518435 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 02:17:54.519216 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":323,"spinlock_wait_cycles":41216}
W20260926 02:17:54.522933 13554 leader_election.cc:341] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359): Not found: Tablet not found: 3c19750a0f72407bb85ddb77e90e4f49
W20260926 02:17:54.523268 13554 leader_election.cc:341] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409): Not found: Tablet not found: 3c19750a0f72407bb85ddb77e90e4f49
I20260926 02:17:54.523394 13554 leader_election.cc:302] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a6b55930f1f49e1b1f120231d0bf80e; no voters: 0d9a4ee5082042d58641ed5ec8b980f3, 4c46720e86fd46c988754878e59a9d61
I20260926 02:17:54.523804 13938 raft_consensus.cc:2726] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:17:54.586220 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:54.585234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:59587)
0926 02:17:54.585735 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:54.585887 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":236,"threads_started":1}
I20260926 02:17:55.038771 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:55.037965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:57193)
0926 02:17:55.038414 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:55.038532 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":224,"threads_started":1}
I20260926 02:17:55.129864 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:55.129094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:51679)
0926 02:17:55.129592 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:55.129679 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":234,"threads_started":1}
I20260926 02:17:55.587967 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:55.587195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:43855)
0926 02:17:55.587680 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:55.587800 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":258,"threads_started":1}
I20260926 02:17:56.040585 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:56.039778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:55823)
0926 02:17:56.040297 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:56.040402 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":285,"threads_started":1}
I20260926 02:17:56.131898 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:56.130893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:55025)
0926 02:17:56.131601 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:56.131696 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":361,"threads_started":1}
I20260926 02:17:56.557449 13948 raft_consensus.cc:482] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:17:56.557654 13948 raft_consensus.cc:504] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:56.558300 13948 leader_election.cc:288] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359), 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409)
W20260926 02:17:56.559372 13554 leader_election.cc:341] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409): Not found: Tablet not found: 3c19750a0f72407bb85ddb77e90e4f49
W20260926 02:17:56.559589 13554 leader_election.cc:341] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359): Not found: Tablet not found: 3c19750a0f72407bb85ddb77e90e4f49
I20260926 02:17:56.559664 13554 leader_election.cc:302] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a6b55930f1f49e1b1f120231d0bf80e; no voters: 0d9a4ee5082042d58641ed5ec8b980f3, 4c46720e86fd46c988754878e59a9d61
I20260926 02:17:56.559880 13948 raft_consensus.cc:2726] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:17:56.589800 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:56.588989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:43145)
0926 02:17:56.589520 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:56.589618 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":287,"threads_started":1}
I20260926 02:17:57.042495 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:57.041680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:60885)
0926 02:17:57.042186 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:57.042281 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":264,"threads_started":1}
I20260926 02:17:57.133749 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:57.132915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:35779)
0926 02:17:57.133480 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:57.133576 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":344,"threads_started":1}
I20260926 02:17:57.591706 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:57.590897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:49177)
0926 02:17:57.591416 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:57.591513 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":252,"threads_started":1}
I20260926 02:17:58.044438 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:58.043598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:49721)
0926 02:17:58.044097 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:58.044190 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":231,"threads_started":1}
I20260926 02:17:58.135664 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:58.134788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:58803)
0926 02:17:58.135299 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:58.135455 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":246,"threads_started":1}
I20260926 02:17:58.593667 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:58.592803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:60033)
0926 02:17:58.593350 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:58.593447 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":267,"threads_started":1}
I20260926 02:17:59.046386 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:59.045618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:43265)
0926 02:17:59.046119 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:59.046220 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
I20260926 02:17:59.137712 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:59.136729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:47661)
0926 02:17:59.137372 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:59.137509 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":335,"threads_started":1}
I20260926 02:17:59.409548 13958 raft_consensus.cc:482] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:17:59.409799 13958 raft_consensus.cc:504] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:17:59.410534 13958 leader_election.cc:288] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359), 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409)
W20260926 02:17:59.411581 13554 leader_election.cc:341] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409): Not found: Tablet not found: 3c19750a0f72407bb85ddb77e90e4f49
W20260926 02:17:59.411773 13554 leader_election.cc:341] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359): Not found: Tablet not found: 3c19750a0f72407bb85ddb77e90e4f49
I20260926 02:17:59.411881 13554 leader_election.cc:302] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a6b55930f1f49e1b1f120231d0bf80e; no voters: 0d9a4ee5082042d58641ed5ec8b980f3, 4c46720e86fd46c988754878e59a9d61
I20260926 02:17:59.412182 13958 raft_consensus.cc:2726] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:17:59.595553 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:17:59.594707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:37945)
0926 02:17:59.595274 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 02:17:59.595367 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":330,"threads_started":1}
I20260926 02:18:00.048153 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:00.047370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:59789)
0926 02:18:00.047862 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:00.047964 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":258,"threads_started":1}
I20260926 02:18:00.139590 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:00.138787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:53909)
0926 02:18:00.139273 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:00.139362 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":256,"threads_started":1}
I20260926 02:18:00.597687 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:00.596713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:55979)
0926 02:18:00.597381 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:00.597491 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":313,"threads_started":1}
I20260926 02:18:01.050182 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:01.049265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:47249)
0926 02:18:01.049858 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:01.049967 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":281,"threads_started":1}
I20260926 02:18:01.141784 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:01.140742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:44521)
0926 02:18:01.141439 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:01.141577 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":413,"threads_started":1}
I20260926 02:18:01.599399 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:01.598629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:35673)
0926 02:18:01.599116 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:01.599199 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":259,"threads_started":1}
I20260926 02:18:02.052075 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:02.051217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:57919)
0926 02:18:02.051717 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:02.051857 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
I20260926 02:18:02.060349 13967 raft_consensus.cc:482] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:18:02.060585 13967 raft_consensus.cc:504] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:18:02.061275 13967 leader_election.cc:288] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359), 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409)
W20260926 02:18:02.062341 13554 leader_election.cc:341] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359): Not found: Tablet not found: 3c19750a0f72407bb85ddb77e90e4f49
W20260926 02:18:02.062618 13554 leader_election.cc:341] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409): Not found: Tablet not found: 3c19750a0f72407bb85ddb77e90e4f49
I20260926 02:18:02.062734 13554 leader_election.cc:302] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a6b55930f1f49e1b1f120231d0bf80e; no voters: 0d9a4ee5082042d58641ed5ec8b980f3, 4c46720e86fd46c988754878e59a9d61
I20260926 02:18:02.063002 13967 raft_consensus.cc:2726] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:18:02.143734 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:02.142797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:45783)
0926 02:18:02.143407 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:02.143538 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":305,"threads_started":1}
I20260926 02:18:02.601279 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:02.600390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:55429)
0926 02:18:02.600956 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:02.601065 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":270,"threads_started":1}
I20260926 02:18:03.053927 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:03.053067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:49265)
0926 02:18:03.053631 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:03.053744 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":253,"threads_started":1}
I20260926 02:18:03.145665 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:03.144856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:52243)
0926 02:18:03.145386 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:03.145483 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":297,"threads_started":1}
I20260926 02:18:03.603118 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:03.602326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:56827)
0926 02:18:03.602821 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:03.602925 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260926 02:18:04.055696 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:04.054967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:49581)
0926 02:18:04.055409 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:04.055502 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":227,"threads_started":1}
I20260926 02:18:04.147485 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:04.146684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:58699)
0926 02:18:04.147181 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:04.147277 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":254,"threads_started":1}
I20260926 02:18:04.605068 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:04.604287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:41187)
0926 02:18:04.604796 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:04.604904 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":253,"threads_started":1}
I20260926 02:18:05.057575 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:05.056684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:37753)
0926 02:18:05.057301 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:05.057401 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":259,"threads_started":1}
I20260926 02:18:05.149353 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:05.148528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:51153)
0926 02:18:05.149043 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:05.149181 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":270,"threads_started":1}
I20260926 02:18:05.209370 13979 raft_consensus.cc:482] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:18:05.209609 13979 raft_consensus.cc:504] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c46720e86fd46c988754878e59a9d61" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 43359 } } peers { permanent_uuid: "2a6b55930f1f49e1b1f120231d0bf80e" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42041 } } peers { permanent_uuid: "0d9a4ee5082042d58641ed5ec8b980f3" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33409 } }
I20260926 02:18:05.210459 13979 leader_election.cc:288] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359), 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409)
W20260926 02:18:05.211464 13554 leader_election.cc:341] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d9a4ee5082042d58641ed5ec8b980f3 (127.28.152.130:33409): Not found: Tablet not found: 3c19750a0f72407bb85ddb77e90e4f49
W20260926 02:18:05.211659 13554 leader_election.cc:341] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c46720e86fd46c988754878e59a9d61 (127.28.152.131:43359): Not found: Tablet not found: 3c19750a0f72407bb85ddb77e90e4f49
I20260926 02:18:05.211751 13554 leader_election.cc:302] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a6b55930f1f49e1b1f120231d0bf80e; no voters: 0d9a4ee5082042d58641ed5ec8b980f3, 4c46720e86fd46c988754878e59a9d61
I20260926 02:18:05.211977 13979 raft_consensus.cc:2726] T 3c19750a0f72407bb85ddb77e90e4f49 P 2a6b55930f1f49e1b1f120231d0bf80e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:18:05.606832 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:05.606078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:37759)
0926 02:18:05.606543 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:05.606630 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":228,"threads_started":1}
I20260926 02:18:06.059499 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:06.058586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:33803)
0926 02:18:06.059231 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:06.059316 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":424,"threads_started":1}
I20260926 02:18:06.151094 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:06.150350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:49039)
0926 02:18:06.150829 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:06.150914 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":257,"threads_started":1}
I20260926 02:18:06.608646 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:06.607829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:53829)
0926 02:18:06.608331 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:06.608467 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":244,"threads_started":1}
I20260926 02:18:07.061491 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:07.060594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:53967)
0926 02:18:07.061199 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:07.061319 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":285,"threads_started":1}
I20260926 02:18:07.152872 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:07.152050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:49137)
0926 02:18:07.152567 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:07.152672 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":294,"threads_started":1}
I20260926 02:18:07.610620 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:07.609712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:49197)
0926 02:18:07.610300 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:07.610397 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":326,"threads_started":1}
I20260926 02:18:08.063257 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:08.062473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:51129)
0926 02:18:08.062972 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:08.063055 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":274,"threads_started":1}
I20260926 02:18:08.154929 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:08.154104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.129:60459)
0926 02:18:08.154646 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:08.154747 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":257,"threads_started":1}
I20260926 02:18:08.612625 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:08.611863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.130:56205)
0926 02:18:08.612345 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:08.612459 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":239,"threads_started":1}
I20260926 02:18:09.035876 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 13498
I20260926 02:18:09.057906 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:09.037085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.28.152.129:42041)
0926 02:18:09.037892 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:09.037902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:09.037935 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:18:09.038027 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:09.038039 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:09.038364 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:09.038562 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:09.040013 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:09.041286 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:09.042912 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:09.043620 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:09.043773 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:09.051898 (+ 8125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:09.051918 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:09.051935 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:09.051982 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:09.055095 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:09.055929 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:09.055938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:09.055941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:09.056090 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:09.056432 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:09.056434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:09.056436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:09.056803 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:09.057040 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:09.057321 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 02:18:09.057539 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":433,"threads_started":1}
I20260926 02:18:09.059280 13664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:09.065411 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:09.064512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:34341 (local address 127.28.152.131:39531)
0926 02:18:09.065056 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:09.065249 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.190:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":293,"threads_started":1}
I20260926 02:18:09.153354 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 13498
I20260926 02:18:09.167600 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 13673
I20260926 02:18:09.180660 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:09.168325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49646 (local address 127.28.152.130:33409)
0926 02:18:09.168823 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:09.168827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:09.168842 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:18:09.168893 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:09.168896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:09.168952 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:09.169046 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:09.170360 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:09.171158 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:09.172388 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:09.172827 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:09.172932 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:09.175606 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:09.175612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:09.175616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:09.175649 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:09.178149 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:09.178800 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:09.178806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:09.178808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:09.178936 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:09.179282 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:09.179286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:09.179288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:09.179540 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:09.179798 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:09.180088 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 02:18:09.180315 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":254,"threads_started":1}
I20260926 02:18:09.181437 13797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:09.267105 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 13673
I20260926 02:18:09.281610 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 13805
I20260926 02:18:09.295540 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:09.282411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.28.152.131:43359)
0926 02:18:09.282983 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:09.282987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:09.283005 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:18:09.283061 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:09.283065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:09.283139 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:09.283231 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:09.284674 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:09.285603 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:09.286942 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:09.287409 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:09.287516 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:09.290151 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:09.290156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:09.290160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:09.290199 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:09.292962 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:09.293675 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:09.293679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:09.293681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:09.293821 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:09.294178 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:09.294182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:09.294184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:09.294477 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:09.294708 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:09.294984 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 02:18:09.295186 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":307,"threads_started":1}
I20260926 02:18:09.296267 13928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:09.379832 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 13805
2026-09-26T02:18:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64239 ms)
[----------] 30 tests from SecurityITest (297703 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:09 dist-test-slave-hllj krb5kdc[14000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:09 dist-test-slave-hllj krb5kdc[14000](info): set up 2 sockets
Sep 26 02:18:09 dist-test-slave-hllj krb5kdc[14000](info): commencing operation
krb5kdc: starting...
W20260926 02:18:10.398962 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:10 dist-test-slave-hllj krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:18:10.657366 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:42873
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:42873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:10.999612 14013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:10.999941 14013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:11.000072 14013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:11.007627 14013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:11.007766 14013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:11.007812 14013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:11.007891 14013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:11.007951 14013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:11.018258 14013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:42873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:42873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:11.019681 14013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:11.021191 14013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:11.033692 14018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:11.033705 14019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:11.033737 14021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:11.034199 14013 server_base.cc:1034] running on GCE node
I20260926 02:18:11.035269 14013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:11.036794 14013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:11.038055 14013 hybrid_clock.cc:648] HybridClock initialized: now 1790389091038007 us; error 38 us; skew 500 ppm
I20260926 02:18:11.038455 14013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:11 dist-test-slave-hllj krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389091, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:11.041579 14013 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:18:11.043136 14013 webserver.cc:460] Webserver started at http://127.28.152.190:39921/ using document root <none> and password file <none>
I20260926 02:18:11.043860 14013 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:11.043982 14013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:11.044291 14013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:11.047540 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "9e7eeaec61d8453a89e25cc4fa074cc2"
format_stamp: "Formatted at 2026-09-26 02:18:11 on dist-test-slave-hllj"
server_key: "8a0092de6e415bb01c09fe1ce9ea4dde"
server_key_iv: "2b281a49de30479348c8c11e9ee877fb"
server_key_version: "encryptionkey@0"
I20260926 02:18:11.048383 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "9e7eeaec61d8453a89e25cc4fa074cc2"
format_stamp: "Formatted at 2026-09-26 02:18:11 on dist-test-slave-hllj"
server_key: "8a0092de6e415bb01c09fe1ce9ea4dde"
server_key_iv: "2b281a49de30479348c8c11e9ee877fb"
server_key_version: "encryptionkey@0"
I20260926 02:18:11.053892 14013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:18:11.057612 14028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:11.059149 14013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:18:11.059329 14013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "9e7eeaec61d8453a89e25cc4fa074cc2"
format_stamp: "Formatted at 2026-09-26 02:18:11 on dist-test-slave-hllj"
server_key: "8a0092de6e415bb01c09fe1ce9ea4dde"
server_key_iv: "2b281a49de30479348c8c11e9ee877fb"
server_key_version: "encryptionkey@0"
I20260926 02:18:11.059527 14013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:11.077307 14013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:11.081667 14013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:11.081944 14013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:11.099843 14013 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:42873
I20260926 02:18:11.099861 14080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:42873 every 8 connection(s)
I20260926 02:18:11.101346 14013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:11.102305 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 14013
I20260926 02:18:11.102535 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:11.103240 29282 external_mini_cluster.cc:1428] Setting key a02ab8f4446b719a3623d436c3c067f4
I20260926 02:18:11.105971 14081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:18:11 dist-test-slave-hllj krb5kdc[14000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:11.119585 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2: Bootstrap starting.
I20260926 02:18:11.122900 14081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:11.123942 14081 log.cc:824] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:11.126948 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2: No bootstrap required, opened a new log
I20260926 02:18:11.127460 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:11.105441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.28.152.190:42873)
0926 02:18:11.106230 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:11.106250 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:11.106298 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:18:11.107939 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:11.107964 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:11.108478 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:11.108859 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:11.110881 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:11.112308 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:11.114195 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:11.114926 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:11.115088 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:11.120057 (+ 4969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:11.120066 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:11.120083 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:11.120124 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:11.123094 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:11.123846 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:11.123853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:11.123860 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:11.124034 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:11.124480 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:11.124485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:11.124487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:11.125077 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:11.125419 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:11.125879 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 02:18:11.126319 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":322,"threads_started":1}
I20260926 02:18:11.135937 14081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7eeaec61d8453a89e25cc4fa074cc2" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42873 } }
I20260926 02:18:11.136371 14081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:11.136472 14081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7eeaec61d8453a89e25cc4fa074cc2, State: Initialized, Role: FOLLOWER
I20260926 02:18:11.137101 14081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7eeaec61d8453a89e25cc4fa074cc2" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42873 } }
I20260926 02:18:11.137406 14081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:11.137543 14081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:11.137718 14081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:11.140035 14081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7eeaec61d8453a89e25cc4fa074cc2" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42873 } }
I20260926 02:18:11.140702 14081 leader_election.cc:302] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e7eeaec61d8453a89e25cc4fa074cc2; no voters:
I20260926 02:18:11.141366 14081 leader_election.cc:288] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:11.141491 14086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:11.142336 14086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [term 1 LEADER]: Becoming Leader. State: Replica: 9e7eeaec61d8453a89e25cc4fa074cc2, State: Running, Role: LEADER
I20260926 02:18:11.142897 14086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7eeaec61d8453a89e25cc4fa074cc2" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42873 } }
I20260926 02:18:11.143546 14081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:11.146165 14087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e7eeaec61d8453a89e25cc4fa074cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7eeaec61d8453a89e25cc4fa074cc2" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42873 } } }
I20260926 02:18:11.146768 14087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:11.148875 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e7eeaec61d8453a89e25cc4fa074cc2. Latest consensus state: current_term: 1 leader_uuid: "9e7eeaec61d8453a89e25cc4fa074cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7eeaec61d8453a89e25cc4fa074cc2" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 42873 } } }
I20260926 02:18:11.149202 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:11.150282 14096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:11.154633 14096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:11.164194 14096 catalog_manager.cc:1347] Generated new cluster ID: a1c670c406f8488d91050662f70a2516
I20260926 02:18:11.164386 14096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:11.178664 14096 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:11.180014 14096 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:11.187897 14096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2: Generated new TSK 0
I20260926 02:18:11.188803 14096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:18:11.250056 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42873
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:11.596829 14107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:11.597150 14107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:11.597280 14107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:11.604934 14107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:11.605103 14107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:11.605278 14107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:18:11.615666 14107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:11.617106 14107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:11.618505 14107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:11.632230 14112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:11.632316 14115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:11.632297 14113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:11.633165 14107 server_base.cc:1034] running on GCE node
I20260926 02:18:11.633822 14107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:11.634824 14107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:11.636066 14107 hybrid_clock.cc:648] HybridClock initialized: now 1790389091636003 us; error 56 us; skew 500 ppm
I20260926 02:18:11.636423 14107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:11 dist-test-slave-hllj krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389091, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:11.639892 14107 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:18:11.641536 14107 webserver.cc:460] Webserver started at http://127.28.152.129:40087/ using document root <none> and password file <none>
I20260926 02:18:11.642266 14107 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:11.642390 14107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:11.642693 14107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:11.645845 14107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "d5c68feefda74a24b7f8aa441233e211"
format_stamp: "Formatted at 2026-09-26 02:18:11 on dist-test-slave-hllj"
server_key: "555df75907acb7843f5671a3a730bcdd"
server_key_iv: "34a836546a55e20c473c76faf0d6e549"
server_key_version: "encryptionkey@0"
I20260926 02:18:11.646677 14107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "d5c68feefda74a24b7f8aa441233e211"
format_stamp: "Formatted at 2026-09-26 02:18:11 on dist-test-slave-hllj"
server_key: "555df75907acb7843f5671a3a730bcdd"
server_key_iv: "34a836546a55e20c473c76faf0d6e549"
server_key_version: "encryptionkey@0"
I20260926 02:18:11.652042 14107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:18:11.655704 14122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:11.657210 14107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:18:11.657380 14107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "d5c68feefda74a24b7f8aa441233e211"
format_stamp: "Formatted at 2026-09-26 02:18:11 on dist-test-slave-hllj"
server_key: "555df75907acb7843f5671a3a730bcdd"
server_key_iv: "34a836546a55e20c473c76faf0d6e549"
server_key_version: "encryptionkey@0"
I20260926 02:18:11.657580 14107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:11.670724 14107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:11.675994 14107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:11.676291 14107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:11.677416 14107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:11.678896 14107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:11.678982 14107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:11.679111 14107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:11.679164 14107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:11.709555 14107 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:45329
I20260926 02:18:11.709605 14235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:45329 every 8 connection(s)
I20260926 02:18:11.710891 14107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:18:11.718677 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 14107
I20260926 02:18:11.718875 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:18:11.719503 29282 external_mini_cluster.cc:1428] Setting key 7f77dd732d869dae157c5b898d1a96f7
Sep 26 02:18:11 dist-test-slave-hllj krb5kdc[14000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389091, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:11.732342 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:11.713685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:35945 (local address 127.28.152.190:42873)
0926 02:18:11.714121 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:11.714126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:11.715303 (+ 1177us) server_negotiation.cc:407] Connection header received
0926 02:18:11.717251 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:11.717257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:11.717357 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:11.717479 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:11.719210 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:11.719945 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:11.721801 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:11.722192 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:11.722299 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:11.726062 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:11.726068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:11.726072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:11.726114 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:11.728640 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:11.729724 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:11.729746 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:11.729750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:11.729915 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:11.730537 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:11.730543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:11.730546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:11.730891 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:11.731046 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:11.731753 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 02:18:11.731986 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":216,"threads_started":1}
I20260926 02:18:11.733314 14239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:11.714265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:42873 (local address 127.28.152.129:35945)
0926 02:18:11.715095 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:11.715142 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:11.716907 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:11.717751 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:11.717761 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:11.718323 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:11.719019 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:11.719041 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:11.720103 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:11.720107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:11.721667 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:11.721680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:11.722414 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:11.722423 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:11.722662 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:11.723301 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:18:11.723325 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:11.725742 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:11.728804 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:11.728815 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:11.728836 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:11.729550 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:11.730051 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:11.730057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:11.730061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:11.730392 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:11.730999 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:11.731005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:11.731378 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 02:18:11.731611 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 02:18:11.732036 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":365,"threads_started":1}
I20260926 02:18:11.734951 14236 heartbeater.cc:347] Connected to a master server at 127.28.152.190:42873
I20260926 02:18:11.735297 14236 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:11.736024 14236 heartbeater.cc:511] Master 127.28.152.190:42873 requested a full tablet report, sending...
I20260926 02:18:11.738204 14045 ts_manager.cc:194] Registered new tserver with Master: d5c68feefda74a24b7f8aa441233e211 (127.28.152.129:45329)
I20260926 02:18:11.739745 14045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:35945
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:18:11.773825 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42873
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:12.111106 14242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:12.111398 14242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:12.111523 14242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:12.119300 14242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:12.119427 14242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:12.119585 14242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:18:12.130163 14242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:12.131600 14242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:12.132988 14242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:12.147296 14250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:12.147323 14248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:12.147320 14247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:12.149050 14242 server_base.cc:1034] running on GCE node
I20260926 02:18:12.149778 14242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:12.150918 14242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:12.152189 14242 hybrid_clock.cc:648] HybridClock initialized: now 1790389092152118 us; error 64 us; skew 500 ppm
I20260926 02:18:12.152547 14242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:12 dist-test-slave-hllj krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389092, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:12.155813 14242 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:18:12.157435 14242 webserver.cc:460] Webserver started at http://127.28.152.130:33331/ using document root <none> and password file <none>
I20260926 02:18:12.158126 14242 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:12.158231 14242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:12.158502 14242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:12.161648 14242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "1fc7f98bbc0e4b4081ec0961df26e48b"
format_stamp: "Formatted at 2026-09-26 02:18:12 on dist-test-slave-hllj"
server_key: "b8d0b056a75698305d247d3d147e6302"
server_key_iv: "b0250d307b9b8e4d7a843159078ce71c"
server_key_version: "encryptionkey@0"
I20260926 02:18:12.162416 14242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "1fc7f98bbc0e4b4081ec0961df26e48b"
format_stamp: "Formatted at 2026-09-26 02:18:12 on dist-test-slave-hllj"
server_key: "b8d0b056a75698305d247d3d147e6302"
server_key_iv: "b0250d307b9b8e4d7a843159078ce71c"
server_key_version: "encryptionkey@0"
I20260926 02:18:12.167997 14242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 02:18:12.171900 14257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:12.173636 14242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:18:12.173843 14242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "1fc7f98bbc0e4b4081ec0961df26e48b"
format_stamp: "Formatted at 2026-09-26 02:18:12 on dist-test-slave-hllj"
server_key: "b8d0b056a75698305d247d3d147e6302"
server_key_iv: "b0250d307b9b8e4d7a843159078ce71c"
server_key_version: "encryptionkey@0"
I20260926 02:18:12.174073 14242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:12.192781 14242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:12.197461 14242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:12.197753 14242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:12.198746 14242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:12.200203 14242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:12.200287 14242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:12.200389 14242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:12.200438 14242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:12.232125 14242 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:46139
I20260926 02:18:12.232177 14370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:46139 every 8 connection(s)
I20260926 02:18:12.233656 14242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:18:12.238184 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 14242
I20260926 02:18:12.238382 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:18:12.238924 29282 external_mini_cluster.cc:1428] Setting key 92fa9a7c8d7cb21a770e57173e544928
Sep 26 02:18:12 dist-test-slave-hllj krb5kdc[14000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389092, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:12.255844 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.236347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:44369 (local address 127.28.152.190:42873)
0926 02:18:12.236829 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:12.236834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:12.238008 (+ 1174us) server_negotiation.cc:407] Connection header received
0926 02:18:12.240408 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:12.240414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:12.240507 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:12.240600 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:12.242346 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.243116 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.245322 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.245835 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.245935 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.249987 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:12.249992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:12.249995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:12.250032 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:12.252604 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:12.253453 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:12.253457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:12.253459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:12.253595 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:12.254033 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:12.254037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:12.254039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:12.254314 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:12.254446 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:12.255210 (+ 764us) server_negotiation.cc:299] Negotiation successful
0926 02:18:12.255453 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":245,"threads_started":1}
I20260926 02:18:12.256904 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.236994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:42873 (local address 127.28.152.130:44369)
0926 02:18:12.237804 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:12.237856 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:12.240124 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:12.240884 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:12.240898 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:12.241562 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:12.242166 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.242185 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.243294 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.243299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.245186 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.245201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.246083 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.246090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.246381 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.247152 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:18:12.247179 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:12.249730 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:12.252764 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:12.252775 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:12.252794 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:12.253322 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:12.253710 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:12.253715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:12.253718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:12.253917 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:12.254436 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:12.254444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:12.254823 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 02:18:12.255087 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 02:18:12.255698 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":294,"threads_started":1}
I20260926 02:18:12.258517 14371 heartbeater.cc:347] Connected to a master server at 127.28.152.190:42873
I20260926 02:18:12.258886 14371 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:12.259666 14371 heartbeater.cc:511] Master 127.28.152.190:42873 requested a full tablet report, sending...
I20260926 02:18:12.261339 14045 ts_manager.cc:194] Registered new tserver with Master: 1fc7f98bbc0e4b4081ec0961df26e48b (127.28.152.130:46139)
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:18:12.262593 14045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:44369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:18:12.296128 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42873
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:12.639585 14377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:12.639889 14377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:12.639985 14377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:12.647473 14377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:12.647609 14377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:12.647825 14377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:18:12.658644 14377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:42873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:12.660006 14377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:12.661417 14377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:12.675424 14383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:12.675859 14382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:12.676061 14385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:12.676808 14377 server_base.cc:1034] running on GCE node
I20260926 02:18:12.677646 14377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:12.678824 14377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:12.680066 14377 hybrid_clock.cc:648] HybridClock initialized: now 1790389092679992 us; error 57 us; skew 500 ppm
I20260926 02:18:12.680410 14377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:12 dist-test-slave-hllj krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389092, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:12.683606 14377 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:18:12.685249 14377 webserver.cc:460] Webserver started at http://127.28.152.131:37729/ using document root <none> and password file <none>
I20260926 02:18:12.685987 14377 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:12.686108 14377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:12.686414 14377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:12.689805 14377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "8495263c61ef4d979e99721760093194"
format_stamp: "Formatted at 2026-09-26 02:18:12 on dist-test-slave-hllj"
server_key: "a1f395d21d285b17fd6ce8a4470070ff"
server_key_iv: "432d367806c5e73ef03f6eb29504b26d"
server_key_version: "encryptionkey@0"
I20260926 02:18:12.690632 14377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "8495263c61ef4d979e99721760093194"
format_stamp: "Formatted at 2026-09-26 02:18:12 on dist-test-slave-hllj"
server_key: "a1f395d21d285b17fd6ce8a4470070ff"
server_key_iv: "432d367806c5e73ef03f6eb29504b26d"
server_key_version: "encryptionkey@0"
I20260926 02:18:12.696177 14377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:18:12.699968 14392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:12.701454 14377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:18:12.701683 14377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "8495263c61ef4d979e99721760093194"
format_stamp: "Formatted at 2026-09-26 02:18:12 on dist-test-slave-hllj"
server_key: "a1f395d21d285b17fd6ce8a4470070ff"
server_key_iv: "432d367806c5e73ef03f6eb29504b26d"
server_key_version: "encryptionkey@0"
I20260926 02:18:12.701890 14377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:12.718290 14377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:12.722644 14377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:12.722922 14377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:12.724007 14377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:12.725543 14377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:12.725643 14377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:12.725769 14377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:12.725823 14377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:12.743153 14236 heartbeater.cc:503] Master 127.28.152.190:42873 was elected leader, sending a full tablet report...
I20260926 02:18:12.758052 14377 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:41577
I20260926 02:18:12.758101 14505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:41577 every 8 connection(s)
I20260926 02:18:12.759392 14377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:18:12.760682 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 14377
I20260926 02:18:12.760857 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:18:12.761333 29282 external_mini_cluster.cc:1428] Setting key 8bd9bff83702713dd746c28e6d2a5ad5
Sep 26 02:18:12 dist-test-slave-hllj krb5kdc[14000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389092, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:12.780751 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.762189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:60549 (local address 127.28.152.190:42873)
0926 02:18:12.762652 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:12.762658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:12.763915 (+ 1257us) server_negotiation.cc:407] Connection header received
0926 02:18:12.766022 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:12.766027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:12.766132 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:12.766282 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:12.767868 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.768649 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.770313 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.770749 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.770862 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.774688 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:12.774694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:12.774697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:12.774736 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:12.777793 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:12.778595 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:12.778599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:12.778602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:12.778772 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:12.779126 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:12.779129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:12.779130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:12.779449 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:12.779577 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:12.780297 (+ 720us) server_negotiation.cc:299] Negotiation successful
0926 02:18:12.780472 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":203,"threads_started":1}
I20260926 02:18:12.781775 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.762732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:42873 (local address 127.28.152.131:60549)
0926 02:18:12.763771 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:12.763817 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:12.765753 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:12.766534 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:12.766544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:12.767083 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:12.767694 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.767719 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.768807 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.768811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.770163 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.770176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.770981 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.770989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.771222 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.771944 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:18:12.771972 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:12.774381 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:12.777938 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:12.777945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:12.777961 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:12.778481 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:12.778853 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:12.778856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:12.778858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:12.779046 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:12.779600 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:12.779607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:12.779955 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 02:18:12.780201 (+ 246us) client_negotiation.cc:240] Negotiation successful
0926 02:18:12.780609 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":380,"threads_started":1}
I20260926 02:18:12.783160 14506 heartbeater.cc:347] Connected to a master server at 127.28.152.190:42873
I20260926 02:18:12.783444 14506 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:12.784217 14506 heartbeater.cc:511] Master 127.28.152.190:42873 requested a full tablet report, sending...
I20260926 02:18:12.785681 14045 ts_manager.cc:194] Registered new tserver with Master: 8495263c61ef4d979e99721760093194 (127.28.152.131:41577)
I20260926 02:18:12.786495 14045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:60549
I20260926 02:18:12.790311 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:18:12.807966 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.794817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.28.152.190:42873)
0926 02:18:12.794990 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:12.794995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:12.795422 (+ 427us) server_negotiation.cc:407] Connection header received
0926 02:18:12.795570 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:12.795575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:12.795657 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:12.795741 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:12.797202 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.797942 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.799233 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.799633 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.799724 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.802425 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:12.802431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:12.802434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:12.802470 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:12.805113 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:12.805864 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:12.805869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:12.805871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:12.805997 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:12.806464 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:12.806468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:12.806470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:12.806888 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:12.807096 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:12.807514 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 02:18:12.807731 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:18:12.812359 14045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:18:12.819669 14045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:18:12.859084 14523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.832562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:41577 (local address 127.0.0.1:60816)
0926 02:18:12.849232 (+ 16670us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:12.849312 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:12.850238 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:12.851355 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:12.851363 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:12.851403 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:12.851783 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.851796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.853835 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.853841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.856211 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.856227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.857935 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.857942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.858160 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.858226 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 02:18:12.858356 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 02:18:12.858585 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16290,"mutex_wait_us":55,"thread_start_us":16758,"threads_started":1}
I20260926 02:18:12.859748 14521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.834979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45329 (local address 127.0.0.1:57054)
0926 02:18:12.850838 (+ 15859us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:12.850859 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:12.851113 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:12.851979 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:12.851984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:12.852009 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:12.852468 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.852481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.854245 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.854250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.856755 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.856764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.858935 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.858942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.859127 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.859157 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:18:12.859254 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 02:18:12.859372 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15738,"mutex_wait_us":265,"thread_start_us":14564,"threads_started":1}
I20260926 02:18:12.860006 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.832730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.28.152.131:41577)
0926 02:18:12.843455 (+ 10725us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:12.843465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:12.849495 (+ 6030us) server_negotiation.cc:407] Connection header received
0926 02:18:12.850505 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:12.850516 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:12.850830 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:12.851032 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:12.852001 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.853662 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.856374 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.857661 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.857935 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.858128 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:12.859499 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0926 02:18:12.859687 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10522,"thread_start_us":10271,"threads_started":1}
I20260926 02:18:12.860210 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.833364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.28.152.130:46139)
0926 02:18:12.851687 (+ 18323us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:12.851695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:12.851723 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:18:12.851810 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:12.851816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:12.852030 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:12.852186 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:12.852829 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.854324 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.856946 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.858085 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.858248 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.859569 (+ 1321us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:12.859627 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 02:18:12.859762 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18155,"thread_start_us":18039,"threads_started":1}
I20260926 02:18:12.860499 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.835645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.28.152.129:45329)
0926 02:18:12.842294 (+ 6649us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:12.842305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:12.850994 (+ 8689us) server_negotiation.cc:407] Connection header received
0926 02:18:12.851308 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:12.851317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:12.851578 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:12.851740 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:12.852694 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.853881 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.856924 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.858426 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.858617 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.858788 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:12.859987 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0926 02:18:12.860161 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6402,"thread_start_us":6197,"threads_started":1}
I20260926 02:18:12.860713 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.833188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:46139 (local address 127.0.0.1:47296)
0926 02:18:12.849588 (+ 16400us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:12.849608 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:12.849816 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:12.852201 (+ 2385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:12.852206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:12.852228 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:12.852640 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.852653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.854441 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.854446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.856749 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.856764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.858570 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.858578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.858755 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.858785 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:18:12.858868 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 02:18:12.859021 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16269,"mutex_wait_us":2,"thread_start_us":17349,"threads_started":1}
I20260926 02:18:12.864236 14441 tablet_service.cc:1467] Processing CreateTablet for tablet 1244e44017f24495a97bd789e38caf6d (DEFAULT_TABLE table=test-table [id=e0aa11c4d9f14b8399f683af08c3bab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:12.864533 14171 tablet_service.cc:1467] Processing CreateTablet for tablet 1244e44017f24495a97bd789e38caf6d (DEFAULT_TABLE table=test-table [id=e0aa11c4d9f14b8399f683af08c3bab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:12.865097 14306 tablet_service.cc:1467] Processing CreateTablet for tablet 1244e44017f24495a97bd789e38caf6d (DEFAULT_TABLE table=test-table [id=e0aa11c4d9f14b8399f683af08c3bab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:12.865741 14441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1244e44017f24495a97bd789e38caf6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:12.865993 14171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1244e44017f24495a97bd789e38caf6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:12.866403 14306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1244e44017f24495a97bd789e38caf6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:12.879019 14525 tablet_bootstrap.cc:492] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: Bootstrap starting.
I20260926 02:18:12.880882 14526 tablet_bootstrap.cc:492] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: Bootstrap starting.
I20260926 02:18:12.881753 14527 tablet_bootstrap.cc:492] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: Bootstrap starting.
I20260926 02:18:12.882532 14525 tablet_bootstrap.cc:654] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:12.883903 14525 log.cc:824] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:12.885339 14527 tablet_bootstrap.cc:654] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:12.886691 14527 log.cc:824] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:12.887413 14526 tablet_bootstrap.cc:654] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:12.888666 14526 log.cc:824] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:12.888870 14525 tablet_bootstrap.cc:492] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: No bootstrap required, opened a new log
I20260926 02:18:12.889233 14525 ts_tablet_manager.cc:1397] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260926 02:18:12.890213 14527 tablet_bootstrap.cc:492] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: No bootstrap required, opened a new log
I20260926 02:18:12.890542 14527 ts_tablet_manager.cc:1397] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 02:18:12.891448 14526 tablet_bootstrap.cc:492] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: No bootstrap required, opened a new log
I20260926 02:18:12.891721 14526 ts_tablet_manager.cc:1397] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 02:18:12.899521 14526 raft_consensus.cc:348] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8495263c61ef4d979e99721760093194" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41577 } } peers { permanent_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46139 } } peers { permanent_uuid: "d5c68feefda74a24b7f8aa441233e211" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45329 } }
I20260926 02:18:12.899955 14526 raft_consensus.cc:374] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:12.900065 14526 raft_consensus.cc:729] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c68feefda74a24b7f8aa441233e211, State: Initialized, Role: FOLLOWER
I20260926 02:18:12.900713 14526 consensus_queue.cc:260] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8495263c61ef4d979e99721760093194" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41577 } } peers { permanent_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46139 } } peers { permanent_uuid: "d5c68feefda74a24b7f8aa441233e211" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45329 } }
I20260926 02:18:12.901019 14525 raft_consensus.cc:348] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8495263c61ef4d979e99721760093194" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41577 } } peers { permanent_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46139 } } peers { permanent_uuid: "d5c68feefda74a24b7f8aa441233e211" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45329 } }
I20260926 02:18:12.901188 14527 raft_consensus.cc:348] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8495263c61ef4d979e99721760093194" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41577 } } peers { permanent_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46139 } } peers { permanent_uuid: "d5c68feefda74a24b7f8aa441233e211" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45329 } }
I20260926 02:18:12.901563 14525 raft_consensus.cc:374] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:12.901607 14527 raft_consensus.cc:374] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:12.901675 14525 raft_consensus.cc:729] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8495263c61ef4d979e99721760093194, State: Initialized, Role: FOLLOWER
I20260926 02:18:12.901698 14527 raft_consensus.cc:729] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc7f98bbc0e4b4081ec0961df26e48b, State: Initialized, Role: FOLLOWER
I20260926 02:18:12.902482 14525 consensus_queue.cc:260] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8495263c61ef4d979e99721760093194" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41577 } } peers { permanent_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46139 } } peers { permanent_uuid: "d5c68feefda74a24b7f8aa441233e211" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45329 } }
I20260926 02:18:12.902489 14527 consensus_queue.cc:260] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8495263c61ef4d979e99721760093194" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41577 } } peers { permanent_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46139 } } peers { permanent_uuid: "d5c68feefda74a24b7f8aa441233e211" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45329 } }
I20260926 02:18:12.902673 14526 ts_tablet_manager.cc:1428] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 02:18:12.904386 14506 heartbeater.cc:503] Master 127.28.152.190:42873 was elected leader, sending a full tablet report...
I20260926 02:18:12.904402 14371 heartbeater.cc:503] Master 127.28.152.190:42873 was elected leader, sending a full tablet report...
I20260926 02:18:12.905061 14527 ts_tablet_manager.cc:1428] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 02:18:12.906488 14525 ts_tablet_manager.cc:1428] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
W20260926 02:18:12.963222 14237 tablet.cc:2369] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:18:12.974771 14533 raft_consensus.cc:482] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:18:12.975073 14533 raft_consensus.cc:504] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8495263c61ef4d979e99721760093194" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41577 } } peers { permanent_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46139 } } peers { permanent_uuid: "d5c68feefda74a24b7f8aa441233e211" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45329 } }
I20260926 02:18:12.976794 14533 leader_election.cc:288] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fc7f98bbc0e4b4081ec0961df26e48b (127.28.152.130:46139), d5c68feefda74a24b7f8aa441233e211 (127.28.152.129:45329)
I20260926 02:18:12.982755 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.977373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:38811 (local address 127.28.152.129:45329)
0926 02:18:12.977556 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:12.977561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:12.977754 (+ 193us) server_negotiation.cc:407] Connection header received
0926 02:18:12.977856 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:12.977861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:12.977942 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:12.978045 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:12.978479 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.979418 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.981017 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.981863 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.982013 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.982094 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:12.982372 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 02:18:12.982502 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:18:12.982775 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.977066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45329 (local address 127.28.152.131:38811)
0926 02:18:12.977531 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:12.977553 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:12.977705 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:12.978085 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:12.978089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:12.978119 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:12.978326 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.978336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.979537 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.979543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.980855 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.980868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.982014 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.982020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.982202 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.982227 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:18:12.982323 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 02:18:12.982466 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292}
I20260926 02:18:12.983395 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.977285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:46139 (local address 127.28.152.131:42005)
0926 02:18:12.978183 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:12.978199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:12.978312 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:12.978655 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:12.978659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:12.978684 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:12.979000 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.979011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.980043 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.980047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.981504 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:12.981517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.982695 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.982701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:12.982857 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.982887 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:18:12.983024 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 02:18:12.983150 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":432,"threads_started":1}
I20260926 02:18:12.983475 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:12.977329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:42005 (local address 127.28.152.130:46139)
0926 02:18:12.977561 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:12.977567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:12.978283 (+ 716us) server_negotiation.cc:407] Connection header received
0926 02:18:12.978460 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:12.978465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:12.978542 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:12.978617 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:12.979162 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.979881 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.981670 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:12.982483 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:12.982628 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:12.982704 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:12.983087 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 02:18:12.983236 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 02:18:12.984052 14191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1244e44017f24495a97bd789e38caf6d" candidate_uuid: "8495263c61ef4d979e99721760093194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c68feefda74a24b7f8aa441233e211" is_pre_election: true
I20260926 02:18:12.984578 14191 raft_consensus.cc:2454] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8495263c61ef4d979e99721760093194 in term 0.
I20260926 02:18:12.984488 14326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1244e44017f24495a97bd789e38caf6d" candidate_uuid: "8495263c61ef4d979e99721760093194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" is_pre_election: true
I20260926 02:18:12.984995 14326 raft_consensus.cc:2454] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8495263c61ef4d979e99721760093194 in term 0.
I20260926 02:18:12.985334 14394 leader_election.cc:302] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8495263c61ef4d979e99721760093194, d5c68feefda74a24b7f8aa441233e211; no voters:
W20260926 02:18:12.985675 14372 tablet.cc:2369] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:18:12.985764 14533 raft_consensus.cc:2781] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:18:12.985904 14533 raft_consensus.cc:482] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:18:12.986002 14533 raft_consensus.cc:3037] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:12.988180 14533 raft_consensus.cc:504] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8495263c61ef4d979e99721760093194" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41577 } } peers { permanent_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46139 } } peers { permanent_uuid: "d5c68feefda74a24b7f8aa441233e211" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45329 } }
I20260926 02:18:12.988806 14533 leader_election.cc:288] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [CANDIDATE]: Term 1 election: Requested vote from peers 1fc7f98bbc0e4b4081ec0961df26e48b (127.28.152.130:46139), d5c68feefda74a24b7f8aa441233e211 (127.28.152.129:45329)
I20260926 02:18:12.989300 14326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1244e44017f24495a97bd789e38caf6d" candidate_uuid: "8495263c61ef4d979e99721760093194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b"
I20260926 02:18:12.989382 14191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1244e44017f24495a97bd789e38caf6d" candidate_uuid: "8495263c61ef4d979e99721760093194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c68feefda74a24b7f8aa441233e211"
I20260926 02:18:12.989531 14326 raft_consensus.cc:3037] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:12.989593 14191 raft_consensus.cc:3037] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:12.992806 14191 raft_consensus.cc:2454] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8495263c61ef4d979e99721760093194 in term 1.
I20260926 02:18:12.992806 14326 raft_consensus.cc:2454] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8495263c61ef4d979e99721760093194 in term 1.
I20260926 02:18:12.993328 14395 leader_election.cc:302] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc7f98bbc0e4b4081ec0961df26e48b, 8495263c61ef4d979e99721760093194; no voters:
I20260926 02:18:12.993692 14533 raft_consensus.cc:2781] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:12.994117 14533 raft_consensus.cc:686] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 1 LEADER]: Becoming Leader. State: Replica: 8495263c61ef4d979e99721760093194, State: Running, Role: LEADER
I20260926 02:18:12.994639 14533 consensus_queue.cc:237] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8495263c61ef4d979e99721760093194" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41577 } } peers { permanent_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46139 } } peers { permanent_uuid: "d5c68feefda74a24b7f8aa441233e211" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45329 } }
I20260926 02:18:12.999094 14043 catalog_manager.cc:5495] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8495263c61ef4d979e99721760093194 (127.28.152.131). New cstate: current_term: 1 leader_uuid: "8495263c61ef4d979e99721760093194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8495263c61ef4d979e99721760093194" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 41577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c68feefda74a24b7f8aa441233e211" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45329 } health_report { overall_health: UNKNOWN } } }
W20260926 02:18:13.010991 14507 tablet.cc:2369] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:18:13.052650 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:13.046943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.28.152.131:41577)
0926 02:18:13.047160 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:13.047165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:13.047186 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:18:13.047396 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:13.047400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:13.047490 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:13.047630 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:18:13.047945 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:13.048755 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:13.050321 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:13.050771 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:13.050880 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:13.051449 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:18:13.052040 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:18:13.052160 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:13.052298 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 02:18:13.052378 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:18:13.061416 14191 raft_consensus.cc:1261] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211 [term 1 FOLLOWER]: Refusing update from remote peer 8495263c61ef4d979e99721760093194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:13.061412 14326 raft_consensus.cc:1261] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b [term 1 FOLLOWER]: Refusing update from remote peer 8495263c61ef4d979e99721760093194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:13.062426 14535 consensus_queue.cc:1035] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c68feefda74a24b7f8aa441233e211" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:13.062788 14533 consensus_queue.cc:1035] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc7f98bbc0e4b4081ec0961df26e48b" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:13.073859 14538 mvcc.cc:205] Tried to move back new op lower bound from 7333433725168803840 to 7333433724909932544. Current Snapshot: MvccSnapshot[applied={T|T < 7333433725168803840}]
I20260926 02:18:13.091074 14043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:13.091459 14043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:13.095695 14043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e7eeaec61d8453a89e25cc4fa074cc2: Sending DeleteTablet for 3 replicas of tablet 1244e44017f24495a97bd789e38caf6d
I20260926 02:18:13.097080 14441 tablet_service.cc:1514] Processing DeleteTablet for tablet 1244e44017f24495a97bd789e38caf6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:13 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:60816
I20260926 02:18:13.097084 14306 tablet_service.cc:1514] Processing DeleteTablet for tablet 1244e44017f24495a97bd789e38caf6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:13 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:47296
I20260926 02:18:13.097227 14171 tablet_service.cc:1514] Processing DeleteTablet for tablet 1244e44017f24495a97bd789e38caf6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:13 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:57054
I20260926 02:18:13.101056 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 14107
I20260926 02:18:13.110419 14549 tablet_replica.cc:330] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: stopping tablet replica
I20260926 02:18:13.111210 14551 tablet_replica.cc:330] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: stopping tablet replica
I20260926 02:18:13.111999 14551 raft_consensus.cc:2229] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:13.112649 14549 raft_consensus.cc:2229] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:13.113397 14551 raft_consensus.cc:2258] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:13.113953 14549 raft_consensus.cc:2258] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:13.116633 14549 ts_tablet_manager.cc:1905] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:13.118933 14551 ts_tablet_manager.cc:1905] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:18:13 dist-test-slave-hllj krb5kdc[14000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:18:13.130671 14551 ts_tablet_manager.cc:1918] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:13.136103 14549 ts_tablet_manager.cc:1918] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:13.136453 14549 log.cc:1196] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/1244e44017f24495a97bd789e38caf6d
I20260926 02:18:13.137123 14549 ts_tablet_manager.cc:1939] T 1244e44017f24495a97bd789e38caf6d P 1fc7f98bbc0e4b4081ec0961df26e48b: Deleting consensus metadata
I20260926 02:18:13.137445 14551 log.cc:1196] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/1244e44017f24495a97bd789e38caf6d
I20260926 02:18:13.138118 14551 ts_tablet_manager.cc:1939] T 1244e44017f24495a97bd789e38caf6d P d5c68feefda74a24b7f8aa441233e211: Deleting consensus metadata
I20260926 02:18:13.140446 14031 catalog_manager.cc:4841] TS 1fc7f98bbc0e4b4081ec0961df26e48b (127.28.152.130:46139): tablet 1244e44017f24495a97bd789e38caf6d (table test-table [id=e0aa11c4d9f14b8399f683af08c3bab7]) successfully deleted
I20260926 02:18:13.140529 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:13.102586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.28.152.129:45329)
0926 02:18:13.102779 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:13.102785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:13.106216 (+ 3431us) server_negotiation.cc:407] Connection header received
0926 02:18:13.106410 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:13.106416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:13.106524 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:13.106632 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:13.112964 (+ 6332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:13.114221 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:13.116440 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:13.117242 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:13.117362 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:13.123529 (+ 6167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:13.123550 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:13.123574 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:13.123650 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:13.127939 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:13.137393 (+ 9454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:13.137407 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:13.137411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:13.137682 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:13.138334 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:13.138340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:13.138343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:13.138881 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:13.139193 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:13.139864 (+ 671us) server_negotiation.cc:299] Negotiation successful
0926 02:18:13.140139 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:18:13.141935 14231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:13.147711 14550 tablet_replica.cc:330] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: stopping tablet replica
I20260926 02:18:13.148216 14550 raft_consensus.cc:2229] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:18:13.148986 14550 raft_consensus.cc:2258] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:13.151412 14550 ts_tablet_manager.cc:1905] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:13.158983 14550 ts_tablet_manager.cc:1918] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:13.159169 14550 log.cc:1196] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/1244e44017f24495a97bd789e38caf6d
I20260926 02:18:13.159713 14550 ts_tablet_manager.cc:1939] T 1244e44017f24495a97bd789e38caf6d P 8495263c61ef4d979e99721760093194: Deleting consensus metadata
I20260926 02:18:13.161748 14029 catalog_manager.cc:4841] TS 8495263c61ef4d979e99721760093194 (127.28.152.131:41577): tablet 1244e44017f24495a97bd789e38caf6d (table test-table [id=e0aa11c4d9f14b8399f683af08c3bab7]) successfully deleted
I20260926 02:18:13.236054 14030 catalog_manager.cc:4841] TS d5c68feefda74a24b7f8aa441233e211 (127.28.152.129:45329): tablet 1244e44017f24495a97bd789e38caf6d (table test-table [id=e0aa11c4d9f14b8399f683af08c3bab7]) successfully deleted
I20260926 02:18:13.236581 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 14107
I20260926 02:18:13.258977 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 14242
Sep 26 02:18:13 dist-test-slave-hllj krb5kdc[14000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:18:13.277576 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:13.259810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.28.152.130:46139)
0926 02:18:13.259996 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:13.260002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:13.260156 (+ 154us) server_negotiation.cc:407] Connection header received
0926 02:18:13.260336 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:13.260340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:13.260409 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:13.260560 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:13.261955 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:13.263210 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:13.264827 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:13.265489 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:13.265643 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:13.270476 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:13.270504 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:13.270527 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:13.270578 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:13.274365 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:13.275154 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:13.275164 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:13.275169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:13.275336 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:13.275731 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:13.275736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:13.275739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:13.276245 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:13.276563 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:13.276815 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 02:18:13.277050 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:18:13.278280 14366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:13.368856 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 14242
I20260926 02:18:13.383327 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 14377
Sep 26 02:18:13 dist-test-slave-hllj krb5kdc[14000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:18:13.402045 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:13.384308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.28.152.131:41577)
0926 02:18:13.384470 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:13.384474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:13.384492 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:18:13.384591 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:13.384595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:13.384662 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:13.384749 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:13.386619 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:13.387429 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:13.389545 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:13.390005 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:13.390109 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:13.395869 (+ 5760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:13.395884 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:13.395900 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:13.395942 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:13.398917 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:13.399904 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:13.399912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:13.399914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:13.400081 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:13.400505 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:13.400508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:13.400510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:13.400865 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:13.401109 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:13.401557 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 02:18:13.401731 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:18:13.402823 14501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:13.523275 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 14377
I20260926 02:18:13.539450 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 14013
I20260926 02:18:13.540632 14076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:13.649276 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 14013
2026-09-26T02:18:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4272 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T02:18:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:13Z Disabled control of system clock
I20260926 02:18:13.721160 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:39873
--webserver_interface=127.28.152.190
--webserver_port=0
--builtin_ntp_servers=127.28.152.148:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:18:14.056650 14560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:14.056967 14560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:14.057073 14560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:14.064683 14560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:14.064819 14560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:14.064883 14560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:14.064961 14560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:14.065028 14560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:14.074823 14560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:39873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:14.076202 14560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:14.078099 14560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:14.090464 14566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:14.090628 14569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:14.090539 14565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:14.091161 14560 server_base.cc:1034] running on GCE node
I20260926 02:18:14.092031 14560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:14.093672 14560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:14.094897 14560 hybrid_clock.cc:648] HybridClock initialized: now 1790389094094834 us; error 61 us; skew 500 ppm
I20260926 02:18:14.095261 14560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:14.101526 14560 webserver.cc:460] Webserver started at http://127.28.152.190:34083/ using document root <none> and password file <none>
I20260926 02:18:14.102305 14560 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:14.102422 14560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:14.102749 14560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:14.105995 14560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "368e62becd5c437d9ac35eda09b44a81"
format_stamp: "Formatted at 2026-09-26 02:18:14 on dist-test-slave-hllj"
server_key: "ea00b94ecf32c5fca2822d5645758135"
server_key_iv: "6dfeec101f0c6ad17b36f82ac8c87166"
server_key_version: "encryptionkey@0"
I20260926 02:18:14.106845 14560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "368e62becd5c437d9ac35eda09b44a81"
format_stamp: "Formatted at 2026-09-26 02:18:14 on dist-test-slave-hllj"
server_key: "ea00b94ecf32c5fca2822d5645758135"
server_key_iv: "6dfeec101f0c6ad17b36f82ac8c87166"
server_key_version: "encryptionkey@0"
I20260926 02:18:14.112447 14560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 02:18:14.116274 14575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:14.117774 14560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:18:14.117954 14560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "368e62becd5c437d9ac35eda09b44a81"
format_stamp: "Formatted at 2026-09-26 02:18:14 on dist-test-slave-hllj"
server_key: "ea00b94ecf32c5fca2822d5645758135"
server_key_iv: "6dfeec101f0c6ad17b36f82ac8c87166"
server_key_version: "encryptionkey@0"
I20260926 02:18:14.118176 14560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:14.138242 14560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:14.139153 14560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:14.139428 14560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:14.157044 14560 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:39873
I20260926 02:18:14.157050 14626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:39873 every 8 connection(s)
I20260926 02:18:14.158444 14560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:14.162392 14627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:14.165323 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 14560
I20260926 02:18:14.165503 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:14.166056 29282 external_mini_cluster.cc:1428] Setting key c02a9364e518efd688a8077c6f5fab1f
I20260926 02:18:14.174588 14627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81: Bootstrap starting.
I20260926 02:18:14.178077 14627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:14.179278 14627 log.cc:824] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:14.180264 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:14.168101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.28.152.190:39873)
0926 02:18:14.169151 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:14.169166 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:14.169213 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:18:14.170907 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:14.170932 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:14.171447 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:14.171854 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:14.172474 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:14.173811 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:14.175265 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:14.176000 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:14.176190 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:14.177513 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:14.177522 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:14.177539 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:14.178131 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:14.178258 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:14.178283 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:14.178396 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:14.178688 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 02:18:14.179040 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":374,"threads_started":1}
I20260926 02:18:14.182785 14627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81: No bootstrap required, opened a new log
I20260926 02:18:14.190603 14627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e62becd5c437d9ac35eda09b44a81" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39873 } }
I20260926 02:18:14.191026 14627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:14.191124 14627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 368e62becd5c437d9ac35eda09b44a81, State: Initialized, Role: FOLLOWER
I20260926 02:18:14.191767 14627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e62becd5c437d9ac35eda09b44a81" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39873 } }
I20260926 02:18:14.191996 14627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:14.192126 14627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:14.192287 14627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:14.194741 14627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e62becd5c437d9ac35eda09b44a81" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39873 } }
I20260926 02:18:14.195254 14627 leader_election.cc:302] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 368e62becd5c437d9ac35eda09b44a81; no voters:
I20260926 02:18:14.195794 14627 leader_election.cc:288] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:14.195928 14632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:14.196609 14632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [term 1 LEADER]: Becoming Leader. State: Replica: 368e62becd5c437d9ac35eda09b44a81, State: Running, Role: LEADER
I20260926 02:18:14.197207 14632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e62becd5c437d9ac35eda09b44a81" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39873 } }
I20260926 02:18:14.198127 14627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:14.200188 14633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "368e62becd5c437d9ac35eda09b44a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e62becd5c437d9ac35eda09b44a81" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39873 } } }
I20260926 02:18:14.200590 14633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:14.200908 14634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 368e62becd5c437d9ac35eda09b44a81. Latest consensus state: current_term: 1 leader_uuid: "368e62becd5c437d9ac35eda09b44a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e62becd5c437d9ac35eda09b44a81" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 39873 } } }
I20260926 02:18:14.201272 14634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:14.207823 14637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:14.213043 14637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:14.223690 14637 catalog_manager.cc:1347] Generated new cluster ID: d0b7ac8208c24215944ade7117182e95
I20260926 02:18:14.223824 14637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:14.257407 14637 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:14.258776 14637 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:14.273175 14637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81: Generated new TSK 0
I20260926 02:18:14.274103 14637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:14.288105 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39873
--builtin_ntp_servers=127.28.152.148:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:18:14.645828 14651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:14.646104 14651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:14.646227 14651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:14.653951 14651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:14.654062 14651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:14.654202 14651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:18:14.663604 14651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:14.664974 14651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:14.666822 14651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:14.679589 14656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:14.679718 14659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:14.680307 14651 server_base.cc:1034] running on GCE node
W20260926 02:18:14.681366 14657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:14.682058 14651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:14.683384 14651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:14.684633 14651 hybrid_clock.cc:648] HybridClock initialized: now 1790389094684553 us; error 69 us; skew 500 ppm
I20260926 02:18:14.685060 14651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:14.687573 14651 webserver.cc:460] Webserver started at http://127.28.152.129:36115/ using document root <none> and password file <none>
I20260926 02:18:14.688335 14651 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:14.688443 14651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:14.688786 14651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:14.691883 14651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "e7de7fa355f54192a50107451145082d"
format_stamp: "Formatted at 2026-09-26 02:18:14 on dist-test-slave-hllj"
server_key: "8e2211fd1636989c3674e5e8ff0f05fa"
server_key_iv: "6e61b16621095cb842b69985c0a3a605"
server_key_version: "encryptionkey@0"
I20260926 02:18:14.692803 14651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "e7de7fa355f54192a50107451145082d"
format_stamp: "Formatted at 2026-09-26 02:18:14 on dist-test-slave-hllj"
server_key: "8e2211fd1636989c3674e5e8ff0f05fa"
server_key_iv: "6e61b16621095cb842b69985c0a3a605"
server_key_version: "encryptionkey@0"
I20260926 02:18:14.698335 14651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:18:14.701946 14665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:14.703337 14651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:14.703502 14651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "e7de7fa355f54192a50107451145082d"
format_stamp: "Formatted at 2026-09-26 02:18:14 on dist-test-slave-hllj"
server_key: "8e2211fd1636989c3674e5e8ff0f05fa"
server_key_iv: "6e61b16621095cb842b69985c0a3a605"
server_key_version: "encryptionkey@0"
I20260926 02:18:14.703704 14651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:14.733384 14651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:14.734311 14651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:14.734604 14651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:14.735649 14651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:14.737241 14651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:14.737329 14651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:14.737417 14651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:14.737469 14651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:14.767702 14651 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:42405
I20260926 02:18:14.767757 14777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:42405 every 8 connection(s)
I20260926 02:18:14.769008 14651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:18:14.776858 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 14651
I20260926 02:18:14.777040 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:18:14.777645 29282 external_mini_cluster.cc:1428] Setting key a4083bd73c1cb2b61c5ecfc2d5252fd0
I20260926 02:18:14.780326 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39873
--builtin_ntp_servers=127.28.152.148:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:18:14.782186 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:14.771605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:34351 (local address 127.28.152.190:39873)
0926 02:18:14.772055 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:14.772059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:14.773384 (+ 1325us) server_negotiation.cc:407] Connection header received
0926 02:18:14.775220 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:14.775227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:14.775318 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:14.775475 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:14.776570 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:14.777656 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:14.779310 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:14.779728 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:14.779841 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:14.781417 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:14.781423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:14.781426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:14.781445 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:14.781495 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:14.781508 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:14.781583 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:14.781826 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 02:18:14.781912 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":230,"threads_started":1}
I20260926 02:18:14.783713 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:14.772184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39873 (local address 127.28.152.129:34351)
0926 02:18:14.773169 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:14.773237 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:14.774907 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:14.775660 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:14.775669 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:14.775769 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:14.776387 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:14.776410 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:14.777845 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:14.777849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:14.779158 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:14.779171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:14.779962 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:14.779969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:14.780212 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:14.781012 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:18:14.781016 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:14.781078 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:18:14.781216 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:14.781598 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:14.781609 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:14.781618 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 02:18:14.781822 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 02:18:14.782292 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":283,"threads_started":1}
I20260926 02:18:14.785455 14778 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39873
I20260926 02:18:14.785815 14778 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:14.786788 14778 heartbeater.cc:511] Master 127.28.152.190:39873 requested a full tablet report, sending...
I20260926 02:18:14.789331 14592 ts_manager.cc:194] Registered new tserver with Master: e7de7fa355f54192a50107451145082d (127.28.152.129:42405)
I20260926 02:18:14.791047 14592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.129:34351
W20260926 02:18:15.115824 14782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:15.116182 14782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:15.116319 14782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:15.124090 14782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:15.124209 14782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:15.124318 14782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:18:15.133808 14782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:15.135125 14782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:15.136950 14782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:15.150418 14788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:15.150475 14787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:15.150540 14790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:15.150961 14782 server_base.cc:1034] running on GCE node
I20260926 02:18:15.151678 14782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:15.152727 14782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:15.153946 14782 hybrid_clock.cc:648] HybridClock initialized: now 1790389095153890 us; error 53 us; skew 500 ppm
I20260926 02:18:15.154284 14782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:15.157063 14782 webserver.cc:460] Webserver started at http://127.28.152.130:33609/ using document root <none> and password file <none>
I20260926 02:18:15.158087 14782 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:15.158229 14782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:15.158663 14782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:15.162853 14782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "a959629b78e94641a0c55ee8e1dedac6"
format_stamp: "Formatted at 2026-09-26 02:18:15 on dist-test-slave-hllj"
server_key: "8adf66ea7a369f8b2ca39a660a3ed1b9"
server_key_iv: "421e770d0bf9a77e3fc34da110a43091"
server_key_version: "encryptionkey@0"
I20260926 02:18:15.163923 14782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "a959629b78e94641a0c55ee8e1dedac6"
format_stamp: "Formatted at 2026-09-26 02:18:15 on dist-test-slave-hllj"
server_key: "8adf66ea7a369f8b2ca39a660a3ed1b9"
server_key_iv: "421e770d0bf9a77e3fc34da110a43091"
server_key_version: "encryptionkey@0"
I20260926 02:18:15.171008 14782 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s
I20260926 02:18:15.175936 14796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:15.177733 14782 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 02:18:15.177892 14782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "a959629b78e94641a0c55ee8e1dedac6"
format_stamp: "Formatted at 2026-09-26 02:18:15 on dist-test-slave-hllj"
server_key: "8adf66ea7a369f8b2ca39a660a3ed1b9"
server_key_iv: "421e770d0bf9a77e3fc34da110a43091"
server_key_version: "encryptionkey@0"
I20260926 02:18:15.178079 14782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:15.194784 14782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:15.195628 14782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:15.195894 14782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:15.196851 14782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:15.198340 14782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:15.198437 14782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:15.198519 14782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:15.198580 14782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:15.229419 14782 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:36039
I20260926 02:18:15.229492 14910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:36039 every 8 connection(s)
I20260926 02:18:15.230754 14782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:18:15.238488 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 14782
I20260926 02:18:15.238637 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:18:15.239034 29282 external_mini_cluster.cc:1428] Setting key a0f54cc0501cb5a10689b04c2014fb93
I20260926 02:18:15.242264 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39873
--builtin_ntp_servers=127.28.152.148:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:18:15.243937 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:15.233330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:46365 (local address 127.28.152.190:39873)
0926 02:18:15.233509 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:15.233514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:15.235005 (+ 1491us) server_negotiation.cc:407] Connection header received
0926 02:18:15.236823 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:15.236831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:15.236903 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:15.237002 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:15.238178 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.238998 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.241003 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.241575 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.241712 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:15.243118 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:15.243124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:15.243130 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:15.243152 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:15.243193 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:15.243208 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:15.243297 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:15.243548 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 02:18:15.243649 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:18:15.244941 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:15.233900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39873 (local address 127.28.152.130:46365)
0926 02:18:15.234855 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:15.234893 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:15.236512 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:15.237258 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:15.237267 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:15.237365 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:15.237993 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:15.238010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.239175 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.239180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:15.240869 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:15.240884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.241812 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.241818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:15.242063 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:15.242765 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:18:15.242768 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:15.242811 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:18:15.242912 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:15.243330 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:15.243339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:15.243346 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:18:15.243528 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 02:18:15.243853 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":263,"threads_started":1}
I20260926 02:18:15.246407 14911 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39873
I20260926 02:18:15.246706 14911 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:15.247496 14911 heartbeater.cc:511] Master 127.28.152.190:39873 requested a full tablet report, sending...
I20260926 02:18:15.248957 14592 ts_manager.cc:194] Registered new tserver with Master: a959629b78e94641a0c55ee8e1dedac6 (127.28.152.130:36039)
I20260926 02:18:15.249794 14592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.130:46365
W20260926 02:18:15.572217 14914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:15.572532 14914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:15.572654 14914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:15.580166 14914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:15.580287 14914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:15.580431 14914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:18:15.589939 14914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:39873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:15.591290 14914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:15.593094 14914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:15.606503 14919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:15.606438 14922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:15.606520 14920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:15.607597 14914 server_base.cc:1034] running on GCE node
I20260926 02:18:15.608305 14914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:15.609469 14914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:15.610707 14914 hybrid_clock.cc:648] HybridClock initialized: now 1790389095610656 us; error 84 us; skew 500 ppm
I20260926 02:18:15.611037 14914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:15.613404 14914 webserver.cc:460] Webserver started at http://127.28.152.131:39515/ using document root <none> and password file <none>
I20260926 02:18:15.614156 14914 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:15.614261 14914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:15.614573 14914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:15.617745 14914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "4eb3e220bfd84061bb933155b087d133"
format_stamp: "Formatted at 2026-09-26 02:18:15 on dist-test-slave-hllj"
server_key: "72eaac1c09bb87c22a79049496d5a7c8"
server_key_iv: "b2e535a33c6cbcf13a541004beb2a70e"
server_key_version: "encryptionkey@0"
I20260926 02:18:15.618562 14914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "4eb3e220bfd84061bb933155b087d133"
format_stamp: "Formatted at 2026-09-26 02:18:15 on dist-test-slave-hllj"
server_key: "72eaac1c09bb87c22a79049496d5a7c8"
server_key_iv: "b2e535a33c6cbcf13a541004beb2a70e"
server_key_version: "encryptionkey@0"
I20260926 02:18:15.624114 14914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:18:15.628178 14928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:15.629827 14914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:18:15.630014 14914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "4eb3e220bfd84061bb933155b087d133"
format_stamp: "Formatted at 2026-09-26 02:18:15 on dist-test-slave-hllj"
server_key: "72eaac1c09bb87c22a79049496d5a7c8"
server_key_iv: "b2e535a33c6cbcf13a541004beb2a70e"
server_key_version: "encryptionkey@0"
I20260926 02:18:15.630256 14914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:15.642397 14914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:15.643265 14914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:15.643560 14914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:15.644562 14914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:15.646145 14914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:15.646230 14914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:15.646312 14914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:15.646365 14914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:15.676219 14914 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:40143
I20260926 02:18:15.676242 15041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:40143 every 8 connection(s)
I20260926 02:18:15.677626 14914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:18:15.686955 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 14914
I20260926 02:18:15.687136 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:18:15.687783 29282 external_mini_cluster.cc:1428] Setting key 58c086362391ade800532ebebcff8de2
I20260926 02:18:15.691605 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:15.680210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:55101 (local address 127.28.152.190:39873)
0926 02:18:15.680384 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:15.680388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:15.681941 (+ 1553us) server_negotiation.cc:407] Connection header received
0926 02:18:15.683935 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:15.683941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:15.684029 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:15.684121 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:15.685360 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.686293 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.688743 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.689213 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.689336 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:15.690882 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:15.690888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:15.690892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:15.690914 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:15.690956 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:15.690971 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:15.691042 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:15.691242 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 02:18:15.691322 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:18:15.692616 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:15.680731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:39873 (local address 127.28.152.131:55101)
0926 02:18:15.681772 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:15.681820 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:15.683622 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:15.684385 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:15.684394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:15.684504 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:15.685115 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:15.685204 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.686466 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.686471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:15.688554 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:15.688572 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.689452 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.689460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:15.689781 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:15.690518 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:18:15.690521 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:15.690571 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:18:15.690685 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:15.691058 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:15.691069 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:15.691075 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:18:15.691237 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 02:18:15.691559 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":301,"threads_started":1}
I20260926 02:18:15.694041 15042 heartbeater.cc:347] Connected to a master server at 127.28.152.190:39873
I20260926 02:18:15.694337 15042 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:15.695070 15042 heartbeater.cc:511] Master 127.28.152.190:39873 requested a full tablet report, sending...
I20260926 02:18:15.696481 14592 ts_manager.cc:194] Registered new tserver with Master: 4eb3e220bfd84061bb933155b087d133 (127.28.152.131:40143)
I20260926 02:18:15.697381 14592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.131:55101
I20260926 02:18:15.703383 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:18:15.714254 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:15.707657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.28.152.190:39873)
0926 02:18:15.707860 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:15.707865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:15.708733 (+ 868us) server_negotiation.cc:407] Connection header received
0926 02:18:15.708871 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:15.708875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:15.708953 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:15.709062 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:15.709688 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.710408 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.711813 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.712222 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.712319 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:15.713526 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:15.713532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:15.713535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:15.713554 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:15.713588 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:15.713600 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:15.713680 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:15.713815 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 02:18:15.713916 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:18:15.718531 14592 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:18:15.725487 14592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:18:15.774675 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:15.739522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43522 (local address 127.28.152.130:36039)
0926 02:18:15.762314 (+ 22792us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:15.762327 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:15.765451 (+ 3124us) server_negotiation.cc:407] Connection header received
0926 02:18:15.765660 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:15.765671 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:15.766002 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:15.766269 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:15.767397 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.768635 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.771403 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.772883 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.773015 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:15.773290 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:15.774114 (+ 824us) server_negotiation.cc:299] Negotiation successful
0926 02:18:15.774314 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22550,"thread_start_us":22828,"threads_started":1}
I20260926 02:18:15.775601 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:15.738874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:36039 (local address 127.0.0.1:43522)
0926 02:18:15.765268 (+ 26394us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:15.765291 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:15.765468 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:15.766541 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:15.766548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:15.766584 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:15.767185 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:15.767199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.768789 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.768793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:15.771178 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:15.771193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.773473 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.773480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:15.773753 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:15.773848 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 02:18:15.775095 (+ 1247us) client_negotiation.cc:240] Negotiation successful
0926 02:18:15.775296 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26224,"mutex_wait_us":30,"thread_start_us":26078,"threads_started":1}
I20260926 02:18:15.776712 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:15.738484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:40143 (local address 127.0.0.1:49438)
0926 02:18:15.763315 (+ 24831us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:15.763375 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:15.763664 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:15.766280 (+ 2616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:15.766289 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:15.766325 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:15.766900 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:15.766917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.769039 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.769044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:15.772506 (+ 3462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:15.772521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.775928 (+ 3407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.775935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:15.776112 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:15.776147 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:18:15.776227 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 02:18:15.776361 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24475,"mutex_wait_us":1,"thread_start_us":28113,"threads_started":1}
I20260926 02:18:15.776897 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:15.738759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.28.152.131:40143)
0926 02:18:15.765686 (+ 26927us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:15.765695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:15.765727 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:18:15.765821 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:15.765826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:15.766007 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:15.766220 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:15.767353 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.768891 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.772763 (+ 3872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.775540 (+ 2777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.776043 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:15.776289 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:15.776369 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 02:18:15.776537 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26728,"thread_start_us":26790,"threads_started":1}
I20260926 02:18:15.778342 15060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:15.739731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32784 (local address 127.28.152.129:42405)
0926 02:18:15.764347 (+ 24616us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:15.764358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:15.769596 (+ 5238us) server_negotiation.cc:407] Connection header received
0926 02:18:15.769702 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:15.769709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:15.769983 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:15.770173 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:15.771261 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.772944 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.775655 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.777151 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.777340 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:15.777578 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:15.777814 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 02:18:15.777984 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24416,"thread_start_us":24278,"threads_started":1}
I20260926 02:18:15.779241 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:15.739144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42405 (local address 127.0.0.1:32784)
0926 02:18:15.769282 (+ 30138us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:15.769307 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:15.769458 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:15.770425 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:15.770430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:15.770455 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:15.770921 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:15.770935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.773150 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.773156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:15.775428 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:15.775446 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:15.777434 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:15.777441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:15.777613 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:15.777657 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 02:18:15.777737 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 02:18:15.777868 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27628,"mutex_wait_us":33,"thread_start_us":23645,"threads_started":1}
I20260926 02:18:15.780292 14844 tablet_service.cc:1467] Processing CreateTablet for tablet 06792dc79ae14cdb81f013c31b8cc1ba (DEFAULT_TABLE table=test-table [id=df701f24533942c69d056213d74e48d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:15.780787 14977 tablet_service.cc:1467] Processing CreateTablet for tablet 06792dc79ae14cdb81f013c31b8cc1ba (DEFAULT_TABLE table=test-table [id=df701f24533942c69d056213d74e48d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:15.782044 14844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06792dc79ae14cdb81f013c31b8cc1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:15.782245 14977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06792dc79ae14cdb81f013c31b8cc1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:15.782444 14713 tablet_service.cc:1467] Processing CreateTablet for tablet 06792dc79ae14cdb81f013c31b8cc1ba (DEFAULT_TABLE table=test-table [id=df701f24533942c69d056213d74e48d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:15.783664 14713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06792dc79ae14cdb81f013c31b8cc1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:15.794298 14778 heartbeater.cc:503] Master 127.28.152.190:39873 was elected leader, sending a full tablet report...
I20260926 02:18:15.799077 15063 tablet_bootstrap.cc:492] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: Bootstrap starting.
I20260926 02:18:15.800447 15061 tablet_bootstrap.cc:492] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: Bootstrap starting.
I20260926 02:18:15.803803 15061 tablet_bootstrap.cc:654] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:15.804911 15063 tablet_bootstrap.cc:654] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:15.804903 15061 log.cc:824] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:15.806301 15063 log.cc:824] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:15.807755 15062 tablet_bootstrap.cc:492] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: Bootstrap starting.
I20260926 02:18:15.810765 15063 tablet_bootstrap.cc:492] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: No bootstrap required, opened a new log
I20260926 02:18:15.810763 15061 tablet_bootstrap.cc:492] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: No bootstrap required, opened a new log
I20260926 02:18:15.811072 15063 ts_tablet_manager.cc:1397] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260926 02:18:15.811072 15061 ts_tablet_manager.cc:1397] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 02:18:15.811740 15062 tablet_bootstrap.cc:654] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:15.812801 15062 log.cc:824] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:15.815614 15062 tablet_bootstrap.cc:492] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: No bootstrap required, opened a new log
I20260926 02:18:15.815893 15062 ts_tablet_manager.cc:1397] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 02:18:15.822088 15061 raft_consensus.cc:348] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb3e220bfd84061bb933155b087d133" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40143 } } peers { permanent_uuid: "a959629b78e94641a0c55ee8e1dedac6" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36039 } } peers { permanent_uuid: "e7de7fa355f54192a50107451145082d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42405 } }
I20260926 02:18:15.822077 15063 raft_consensus.cc:348] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb3e220bfd84061bb933155b087d133" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40143 } } peers { permanent_uuid: "a959629b78e94641a0c55ee8e1dedac6" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36039 } } peers { permanent_uuid: "e7de7fa355f54192a50107451145082d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42405 } }
I20260926 02:18:15.822687 15063 raft_consensus.cc:374] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:15.822667 15061 raft_consensus.cc:374] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:15.822808 15063 raft_consensus.cc:729] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a959629b78e94641a0c55ee8e1dedac6, State: Initialized, Role: FOLLOWER
I20260926 02:18:15.822805 15061 raft_consensus.cc:729] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7de7fa355f54192a50107451145082d, State: Initialized, Role: FOLLOWER
I20260926 02:18:15.823570 15061 consensus_queue.cc:260] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb3e220bfd84061bb933155b087d133" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40143 } } peers { permanent_uuid: "a959629b78e94641a0c55ee8e1dedac6" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36039 } } peers { permanent_uuid: "e7de7fa355f54192a50107451145082d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42405 } }
I20260926 02:18:15.823549 15063 consensus_queue.cc:260] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb3e220bfd84061bb933155b087d133" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40143 } } peers { permanent_uuid: "a959629b78e94641a0c55ee8e1dedac6" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36039 } } peers { permanent_uuid: "e7de7fa355f54192a50107451145082d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42405 } }
I20260926 02:18:15.823939 15062 raft_consensus.cc:348] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb3e220bfd84061bb933155b087d133" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40143 } } peers { permanent_uuid: "a959629b78e94641a0c55ee8e1dedac6" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36039 } } peers { permanent_uuid: "e7de7fa355f54192a50107451145082d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42405 } }
I20260926 02:18:15.824486 15062 raft_consensus.cc:374] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:15.825407 14911 heartbeater.cc:503] Master 127.28.152.190:39873 was elected leader, sending a full tablet report...
I20260926 02:18:15.825627 15061 ts_tablet_manager.cc:1428] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260926 02:18:15.825950 15063 ts_tablet_manager.cc:1428] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 02:18:15.826125 15062 raft_consensus.cc:729] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3e220bfd84061bb933155b087d133, State: Initialized, Role: FOLLOWER
I20260926 02:18:15.826856 15062 consensus_queue.cc:260] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb3e220bfd84061bb933155b087d133" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40143 } } peers { permanent_uuid: "a959629b78e94641a0c55ee8e1dedac6" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36039 } } peers { permanent_uuid: "e7de7fa355f54192a50107451145082d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42405 } }
I20260926 02:18:15.828048 15042 heartbeater.cc:503] Master 127.28.152.190:39873 was elected leader, sending a full tablet report...
I20260926 02:18:15.828651 15062 ts_tablet_manager.cc:1428] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
W20260926 02:18:15.929304 15043 tablet.cc:2369] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:18:15.982731 14912 tablet.cc:2369] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:18:16.021445 14779 tablet.cc:2369] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:18:16.045562 15068 raft_consensus.cc:482] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:18:16.045893 15068 raft_consensus.cc:504] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb3e220bfd84061bb933155b087d133" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40143 } } peers { permanent_uuid: "a959629b78e94641a0c55ee8e1dedac6" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36039 } } peers { permanent_uuid: "e7de7fa355f54192a50107451145082d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42405 } }
I20260926 02:18:16.047633 15068 leader_election.cc:288] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3e220bfd84061bb933155b087d133 (127.28.152.131:40143), e7de7fa355f54192a50107451145082d (127.28.152.129:42405)
I20260926 02:18:16.054641 15060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.048120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:60337 (local address 127.28.152.129:42405)
0926 02:18:16.048381 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:16.048385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:16.049169 (+ 784us) server_negotiation.cc:407] Connection header received
0926 02:18:16.049351 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:16.049354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:16.049424 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:16.049508 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:16.050123 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.050813 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.052850 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.053676 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.053930 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:16.054034 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:16.054231 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 02:18:16.054367 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260926 02:18:16.054659 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.048196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:37047 (local address 127.28.152.131:40143)
0926 02:18:16.048464 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:16.048471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:16.049067 (+ 596us) server_negotiation.cc:407] Connection header received
0926 02:18:16.049303 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:16.049309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:16.049379 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:16.049453 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:16.050044 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.050770 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.052814 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.053680 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.053843 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:16.054035 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:16.054316 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 02:18:16.054452 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 02:18:16.054728 15071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.048106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42405 (local address 127.28.152.130:60337)
0926 02:18:16.049037 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:16.049054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:16.049217 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:16.049545 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:16.049550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:16.049579 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:16.049972 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:16.049985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.050948 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.050956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:16.052654 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:16.052670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.053891 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.053897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:16.054081 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:16.054109 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 02:18:16.054183 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 02:18:16.054301 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":68,"thread_start_us":644,"threads_started":1}
I20260926 02:18:16.054773 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.047963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:40143 (local address 127.28.152.130:37047)
0926 02:18:16.048939 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:16.048959 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:16.049122 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:16.049493 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:16.049497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:16.049521 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:16.049902 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:16.049911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.050896 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.050902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:16.052614 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:16.052628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.053976 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.053984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:16.054151 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:16.054182 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:18:16.054272 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 02:18:16.054393 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":163,"thread_start_us":603,"threads_started":1}
I20260926 02:18:16.055755 14997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06792dc79ae14cdb81f013c31b8cc1ba" candidate_uuid: "a959629b78e94641a0c55ee8e1dedac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3e220bfd84061bb933155b087d133" is_pre_election: true
I20260926 02:18:16.055774 14733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06792dc79ae14cdb81f013c31b8cc1ba" candidate_uuid: "a959629b78e94641a0c55ee8e1dedac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7de7fa355f54192a50107451145082d" is_pre_election: true
I20260926 02:18:16.056250 14733 raft_consensus.cc:2454] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a959629b78e94641a0c55ee8e1dedac6 in term 0.
I20260926 02:18:16.056249 14997 raft_consensus.cc:2454] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a959629b78e94641a0c55ee8e1dedac6 in term 0.
I20260926 02:18:16.056882 14798 leader_election.cc:302] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a959629b78e94641a0c55ee8e1dedac6, e7de7fa355f54192a50107451145082d; no voters:
I20260926 02:18:16.057322 15068 raft_consensus.cc:2781] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:18:16.057458 15068 raft_consensus.cc:482] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:18:16.057559 15068 raft_consensus.cc:3037] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:16.059835 15068 raft_consensus.cc:504] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb3e220bfd84061bb933155b087d133" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40143 } } peers { permanent_uuid: "a959629b78e94641a0c55ee8e1dedac6" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36039 } } peers { permanent_uuid: "e7de7fa355f54192a50107451145082d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42405 } }
I20260926 02:18:16.060556 15068 leader_election.cc:288] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [CANDIDATE]: Term 1 election: Requested vote from peers 4eb3e220bfd84061bb933155b087d133 (127.28.152.131:40143), e7de7fa355f54192a50107451145082d (127.28.152.129:42405)
I20260926 02:18:16.061106 14997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06792dc79ae14cdb81f013c31b8cc1ba" candidate_uuid: "a959629b78e94641a0c55ee8e1dedac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3e220bfd84061bb933155b087d133"
I20260926 02:18:16.061211 14733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06792dc79ae14cdb81f013c31b8cc1ba" candidate_uuid: "a959629b78e94641a0c55ee8e1dedac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7de7fa355f54192a50107451145082d"
I20260926 02:18:16.061369 14997 raft_consensus.cc:3037] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:16.061425 14733 raft_consensus.cc:3037] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:16.063571 14997 raft_consensus.cc:2454] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a959629b78e94641a0c55ee8e1dedac6 in term 1.
I20260926 02:18:16.063670 14733 raft_consensus.cc:2454] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a959629b78e94641a0c55ee8e1dedac6 in term 1.
I20260926 02:18:16.064049 14799 leader_election.cc:302] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3e220bfd84061bb933155b087d133, a959629b78e94641a0c55ee8e1dedac6; no voters:
I20260926 02:18:16.064384 15068 raft_consensus.cc:2781] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:16.064812 15068 raft_consensus.cc:686] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 1 LEADER]: Becoming Leader. State: Replica: a959629b78e94641a0c55ee8e1dedac6, State: Running, Role: LEADER
I20260926 02:18:16.065341 15068 consensus_queue.cc:237] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb3e220bfd84061bb933155b087d133" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40143 } } peers { permanent_uuid: "a959629b78e94641a0c55ee8e1dedac6" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36039 } } peers { permanent_uuid: "e7de7fa355f54192a50107451145082d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42405 } }
I20260926 02:18:16.070050 14592 catalog_manager.cc:5495] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a959629b78e94641a0c55ee8e1dedac6 (127.28.152.130). New cstate: current_term: 1 leader_uuid: "a959629b78e94641a0c55ee8e1dedac6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb3e220bfd84061bb933155b087d133" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a959629b78e94641a0c55ee8e1dedac6" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7de7fa355f54192a50107451145082d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20260926 02:18:16.150316 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.144727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.28.152.130:36039)
0926 02:18:16.144907 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:16.144912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:16.144945 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:18:16.145040 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:16.145043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:16.145184 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:16.145335 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:18:16.145747 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.146474 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.148079 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.148501 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.148614 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:16.149124 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:18:16.149713 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:18:16.149851 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:16.149967 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 02:18:16.150049 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:18:16.158798 14733 raft_consensus.cc:1261] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d [term 1 FOLLOWER]: Refusing update from remote peer a959629b78e94641a0c55ee8e1dedac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:16.158846 14997 raft_consensus.cc:1261] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133 [term 1 FOLLOWER]: Refusing update from remote peer a959629b78e94641a0c55ee8e1dedac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:16.159845 15068 consensus_queue.cc:1035] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7de7fa355f54192a50107451145082d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:16.160166 15072 consensus_queue.cc:1035] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3e220bfd84061bb933155b087d133" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 40143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:16.177493 15076 mvcc.cc:205] Tried to move back new op lower bound from 7333433737855803392 to 7333433737488216064. Current Snapshot: MvccSnapshot[applied={T|T < 7333433737855803392}]
I20260926 02:18:16.191186 14592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:16.191473 14592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:16.195361 14592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 368e62becd5c437d9ac35eda09b44a81: Sending DeleteTablet for 3 replicas of tablet 06792dc79ae14cdb81f013c31b8cc1ba
I20260926 02:18:16.196606 14977 tablet_service.cc:1514] Processing DeleteTablet for tablet 06792dc79ae14cdb81f013c31b8cc1ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:16 UTC) from {username='slave'} at 127.0.0.1:49438
I20260926 02:18:16.196681 14844 tablet_service.cc:1514] Processing DeleteTablet for tablet 06792dc79ae14cdb81f013c31b8cc1ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:16 UTC) from {username='slave'} at 127.0.0.1:43522
I20260926 02:18:16.196848 14713 tablet_service.cc:1514] Processing DeleteTablet for tablet 06792dc79ae14cdb81f013c31b8cc1ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:16 UTC) from {username='slave'} at 127.0.0.1:32784
I20260926 02:18:16.198457 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 14651
I20260926 02:18:16.210680 15085 tablet_replica.cc:330] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: stopping tablet replica
I20260926 02:18:16.211503 15085 raft_consensus.cc:2229] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:18:16.212491 15085 raft_consensus.cc:2258] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:16.213092 15086 tablet_replica.cc:330] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: stopping tablet replica
I20260926 02:18:16.213459 15084 tablet_replica.cc:330] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: stopping tablet replica
I20260926 02:18:16.213820 15086 raft_consensus.cc:2229] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:16.213894 15084 raft_consensus.cc:2229] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:16.214464 15084 raft_consensus.cc:2258] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:16.214778 15086 raft_consensus.cc:2258] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:16.216910 15085 ts_tablet_manager.cc:1905] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:16.216904 15086 ts_tablet_manager.cc:1905] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:16.221788 15060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.200042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.28.152.129:42405)
0926 02:18:16.200229 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:16.200235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:16.206974 (+ 6739us) server_negotiation.cc:407] Connection header received
0926 02:18:16.207219 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:16.207226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:16.207352 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:16.207488 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:16.213417 (+ 5929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.214705 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.217431 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.218141 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.218259 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:16.219886 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:16.219899 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:16.219928 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:16.220737 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:16.220820 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:16.220851 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:16.220968 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:16.221260 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 02:18:16.221393 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:18:16.223357 14773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:16.224649 15085 ts_tablet_manager.cc:1918] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:16.224936 15085 log.cc:1196] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/06792dc79ae14cdb81f013c31b8cc1ba
I20260926 02:18:16.225654 15085 ts_tablet_manager.cc:1939] T 06792dc79ae14cdb81f013c31b8cc1ba P a959629b78e94641a0c55ee8e1dedac6: Deleting consensus metadata
I20260926 02:18:16.226516 15084 ts_tablet_manager.cc:1905] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:16.227761 14576 catalog_manager.cc:4841] TS a959629b78e94641a0c55ee8e1dedac6 (127.28.152.130:36039): tablet 06792dc79ae14cdb81f013c31b8cc1ba (table test-table [id=df701f24533942c69d056213d74e48d4]) successfully deleted
I20260926 02:18:16.232430 15084 ts_tablet_manager.cc:1918] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:16.232591 15084 log.cc:1196] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/06792dc79ae14cdb81f013c31b8cc1ba
I20260926 02:18:16.233006 15084 ts_tablet_manager.cc:1939] T 06792dc79ae14cdb81f013c31b8cc1ba P 4eb3e220bfd84061bb933155b087d133: Deleting consensus metadata
I20260926 02:18:16.235312 14578 catalog_manager.cc:4841] TS 4eb3e220bfd84061bb933155b087d133 (127.28.152.131:40143): tablet 06792dc79ae14cdb81f013c31b8cc1ba (table test-table [id=df701f24533942c69d056213d74e48d4]) successfully deleted
I20260926 02:18:16.333446 15086 ts_tablet_manager.cc:1918] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:16.333686 15086 log.cc:1196] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/06792dc79ae14cdb81f013c31b8cc1ba
I20260926 02:18:16.334317 15086 ts_tablet_manager.cc:1939] T 06792dc79ae14cdb81f013c31b8cc1ba P e7de7fa355f54192a50107451145082d: Deleting consensus metadata
I20260926 02:18:16.334645 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 14651
I20260926 02:18:16.355759 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 14782
W20260926 02:18:16.355813 14577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.152.129:42405 (error 108)
I20260926 02:18:16.358826 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.357548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42405 (local address 127.0.0.1:32812)
0926 02:18:16.358447 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:16.358659 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":700,"threads_started":1}
W20260926 02:18:16.359123 14577 catalog_manager.cc:4584] TS e7de7fa355f54192a50107451145082d (127.28.152.129:42405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06792dc79ae14cdb81f013c31b8cc1ba: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42405: connect: Connection refused (error 111)
I20260926 02:18:16.364462 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.356633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.28.152.130:36039)
0926 02:18:16.357288 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:16.357294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:16.357312 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:18:16.357375 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:16.357381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:16.357497 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:16.357611 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:16.358155 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.359329 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.360659 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.361566 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.361725 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:16.362772 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:16.362785 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:16.362809 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:16.363631 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:16.363693 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:16.363717 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:16.363830 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:16.363976 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 02:18:16.364083 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521}
I20260926 02:18:16.365293 14906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:16.412752 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.412248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42405 (local address 127.0.0.1:32828)
0926 02:18:16.412492 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:16.412592 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 02:18:16.413841 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.413462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42405 (local address 127.0.0.1:32836)
0926 02:18:16.413601 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:16.413699 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 02:18:16.474560 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.474044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42405 (local address 127.0.0.1:32850)
0926 02:18:16.474302 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:16.474403 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 02:18:16.475595 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.475252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42405 (local address 127.0.0.1:32860)
0926 02:18:16.475376 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:16.475449 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 02:18:16.478868 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 14782
I20260926 02:18:16.495054 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 14914
I20260926 02:18:16.501524 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.495796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.28.152.131:40143)
0926 02:18:16.495961 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:16.495965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:16.495984 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:18:16.496133 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:16.496137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:16.496232 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:16.496318 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:16.496691 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.497433 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.498729 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:16.499133 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:16.499219 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:16.500201 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:16.500210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:16.500224 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:16.500798 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:16.500852 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:16.500872 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:16.500966 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:16.501049 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 02:18:16.501207 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:18:16.502146 15037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:16.556361 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.555871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42405 (local address 127.0.0.1:32876)
0926 02:18:16.556103 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:16.556207 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 02:18:16.557514 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:16.557040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:42405 (local address 127.0.0.1:32892)
0926 02:18:16.557284 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:16.557366 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260926 02:18:16.590152 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 14914
I20260926 02:18:16.604243 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 14560
I20260926 02:18:16.605396 14622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:16.714555 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 14560
2026-09-26T02:18:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3067 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:16 dist-test-slave-hllj krb5kdc[15096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:16 dist-test-slave-hllj krb5kdc[15096](info): set up 2 sockets
Sep 26 02:18:16 dist-test-slave-hllj krb5kdc[15096](info): commencing operation
krb5kdc: starting...
W20260926 02:18:17.746532 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:17 dist-test-slave-hllj krb5kdc[15096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:18:17.996299 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:45409
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:45409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:18.328274 15109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:18.328644 15109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:18.328789 15109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:18.336351 15109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:18.336508 15109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:18.336593 15109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:18.336674 15109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:18.336733 15109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:18.347272 15109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:45409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:45409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:18.348763 15109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:18.350189 15109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:18.362404 15115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:18.362373 15117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:18.362447 15114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:18.363384 15109 server_base.cc:1034] running on GCE node
I20260926 02:18:18.364248 15109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:18.365931 15109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:18.367188 15109 hybrid_clock.cc:648] HybridClock initialized: now 1790389098367113 us; error 60 us; skew 500 ppm
I20260926 02:18:18.367547 15109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:18 dist-test-slave-hllj krb5kdc[15096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:18.370875 15109 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:18:18.372609 15109 webserver.cc:460] Webserver started at http://127.28.152.190:36641/ using document root <none> and password file <none>
I20260926 02:18:18.373381 15109 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:18.373495 15109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:18.373817 15109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:18.376952 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "6f37f6f2d0fb494f9353d1dcf9819dfc"
format_stamp: "Formatted at 2026-09-26 02:18:18 on dist-test-slave-hllj"
server_key: "e951ead4c2bf05b6e4234434d49977f3"
server_key_iv: "f7f3e264a60292080b4b827d0e670d68"
server_key_version: "encryptionkey@0"
I20260926 02:18:18.377815 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "6f37f6f2d0fb494f9353d1dcf9819dfc"
format_stamp: "Formatted at 2026-09-26 02:18:18 on dist-test-slave-hllj"
server_key: "e951ead4c2bf05b6e4234434d49977f3"
server_key_iv: "f7f3e264a60292080b4b827d0e670d68"
server_key_version: "encryptionkey@0"
I20260926 02:18:18.383605 15109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:18:18.387738 15124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:18.389549 15109 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:18:18.389756 15109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "6f37f6f2d0fb494f9353d1dcf9819dfc"
format_stamp: "Formatted at 2026-09-26 02:18:18 on dist-test-slave-hllj"
server_key: "e951ead4c2bf05b6e4234434d49977f3"
server_key_iv: "f7f3e264a60292080b4b827d0e670d68"
server_key_version: "encryptionkey@0"
I20260926 02:18:18.390019 15109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:18.409755 15109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:18.414724 15109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:18.415024 15109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:18.433949 15109 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:45409
I20260926 02:18:18.433955 15175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:45409 every 8 connection(s)
I20260926 02:18:18.435495 15109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:18.439518 15176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:18.440488 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 15109
I20260926 02:18:18.440685 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:18.441365 29282 external_mini_cluster.cc:1428] Setting key c37bc0fee8952f9cce096e1efeb35dd9
I20260926 02:18:18.454296 15176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc: Bootstrap starting.
Sep 26 02:18:18 dist-test-slave-hllj krb5kdc[15096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:18.458546 15176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:18.459841 15176 log.cc:824] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:18.463265 15176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc: No bootstrap required, opened a new log
I20260926 02:18:18.467129 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:18.443321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.28.152.190:45409)
0926 02:18:18.444102 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:18.444115 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:18.444160 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:18:18.446028 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:18.446057 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:18.446551 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:18.446939 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:18.448873 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:18.450370 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:18.452356 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:18.453205 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:18.453358 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:18.458628 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:18.458639 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:18.458659 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:18.458716 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:18.462269 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:18.463350 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:18.463358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:18.463365 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:18.463615 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:18.464106 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:18.464109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:18.464112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:18.464747 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:18.464992 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:18.465548 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 02:18:18.466006 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":352,"threads_started":1}
I20260926 02:18:18.471721 15176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f37f6f2d0fb494f9353d1dcf9819dfc" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45409 } }
I20260926 02:18:18.472159 15176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:18.472260 15176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f37f6f2d0fb494f9353d1dcf9819dfc, State: Initialized, Role: FOLLOWER
I20260926 02:18:18.472857 15176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f37f6f2d0fb494f9353d1dcf9819dfc" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45409 } }
I20260926 02:18:18.473110 15176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:18.473278 15176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:18.473443 15176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:18.475777 15176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f37f6f2d0fb494f9353d1dcf9819dfc" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45409 } }
I20260926 02:18:18.476343 15176 leader_election.cc:302] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f37f6f2d0fb494f9353d1dcf9819dfc; no voters:
I20260926 02:18:18.476975 15176 leader_election.cc:288] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:18.477164 15181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:18.477871 15181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [term 1 LEADER]: Becoming Leader. State: Replica: 6f37f6f2d0fb494f9353d1dcf9819dfc, State: Running, Role: LEADER
I20260926 02:18:18.478454 15181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f37f6f2d0fb494f9353d1dcf9819dfc" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45409 } }
I20260926 02:18:18.479136 15176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:18.481751 15183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f37f6f2d0fb494f9353d1dcf9819dfc. Latest consensus state: current_term: 1 leader_uuid: "6f37f6f2d0fb494f9353d1dcf9819dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f37f6f2d0fb494f9353d1dcf9819dfc" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45409 } } }
I20260926 02:18:18.482123 15182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f37f6f2d0fb494f9353d1dcf9819dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f37f6f2d0fb494f9353d1dcf9819dfc" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45409 } } }
I20260926 02:18:18.482601 15183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:18.483667 15182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:18.490123 15190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:18.495731 15190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:18.505790 15190 catalog_manager.cc:1347] Generated new cluster ID: 1c254d13f71346b4a0b1457e86efb2fb
I20260926 02:18:18.505915 15190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:18.516019 15190 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:18.517429 15190 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:18.523046 15190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc: Generated new TSK 0
I20260926 02:18:18.523995 15190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:18:18.592676 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45409
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:18.930058 15203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:18.930373 15203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:18.930469 15203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:18.937908 15203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:18.938042 15203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:18.938159 15203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:18:18.948371 15203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:18.949710 15203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:18.951092 15203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:18.965075 15209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:18.965166 15211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:18.965274 15208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:18.966506 15203 server_base.cc:1034] running on GCE node
I20260926 02:18:18.967185 15203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:18.968243 15203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:18.969445 15203 hybrid_clock.cc:648] HybridClock initialized: now 1790389098969401 us; error 38 us; skew 500 ppm
I20260926 02:18:18.969779 15203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:18 dist-test-slave-hllj krb5kdc[15096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:18.973026 15203 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:18:18.974682 15203 webserver.cc:460] Webserver started at http://127.28.152.129:44591/ using document root <none> and password file <none>
I20260926 02:18:18.975387 15203 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:18.975493 15203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:18.975773 15203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:18.978855 15203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "21b8fb95c5004e4da70c3228af24c4f2"
format_stamp: "Formatted at 2026-09-26 02:18:18 on dist-test-slave-hllj"
server_key: "e54f300fe79214253612000dc089f306"
server_key_iv: "33fe7a36f5e92c89d7aab85b65603122"
server_key_version: "encryptionkey@0"
I20260926 02:18:18.979687 15203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "21b8fb95c5004e4da70c3228af24c4f2"
format_stamp: "Formatted at 2026-09-26 02:18:18 on dist-test-slave-hllj"
server_key: "e54f300fe79214253612000dc089f306"
server_key_iv: "33fe7a36f5e92c89d7aab85b65603122"
server_key_version: "encryptionkey@0"
I20260926 02:18:18.985718 15203 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:18:18.989593 15218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:18.991258 15203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:18.991407 15203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "21b8fb95c5004e4da70c3228af24c4f2"
format_stamp: "Formatted at 2026-09-26 02:18:18 on dist-test-slave-hllj"
server_key: "e54f300fe79214253612000dc089f306"
server_key_iv: "33fe7a36f5e92c89d7aab85b65603122"
server_key_version: "encryptionkey@0"
I20260926 02:18:18.991611 15203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:19.009387 15203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:19.013831 15203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:19.014093 15203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:19.015084 15203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:19.016477 15203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:19.016549 15203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:19.016647 15203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:19.016688 15203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:19.047457 15203 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:45485
I20260926 02:18:19.047530 15330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:45485 every 8 connection(s)
I20260926 02:18:19.048785 15203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:18:19.057168 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 15203
I20260926 02:18:19.057359 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:18:19.057897 29282 external_mini_cluster.cc:1428] Setting key cf651a25cdb83e0f1c382a27eaa3d92c
Sep 26 02:18:19 dist-test-slave-hllj krb5kdc[15096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:19.070248 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:19.051534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:56525 (local address 127.28.152.190:45409)
0926 02:18:19.052003 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:19.052007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:19.052991 (+ 984us) server_negotiation.cc:407] Connection header received
0926 02:18:19.054865 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:19.054870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:19.054961 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:19.055055 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:19.056623 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:19.057693 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:19.059777 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:19.060275 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:19.060401 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:19.064264 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:19.064270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:19.064274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:19.064312 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:19.066946 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:19.067771 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:19.067776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:19.067779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:19.067922 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:19.068373 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:19.068377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:19.068379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:19.068656 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:19.068794 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:19.069625 (+ 831us) server_negotiation.cc:299] Negotiation successful
0926 02:18:19.069873 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":215,"threads_started":1}
I20260926 02:18:19.071329 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:19.052102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45409 (local address 127.28.152.129:56525)
0926 02:18:19.052810 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:19.052861 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:19.054561 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:19.055316 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:19.055326 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:19.055878 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:19.056453 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:19.056470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:19.057871 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:19.057876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:19.059587 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:19.059605 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:19.060529 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:19.060537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:19.060790 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:19.061491 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:18:19.061519 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:19.063974 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:19.067120 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:19.067131 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:19.067153 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:19.067629 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:19.068038 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:19.068044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:19.068048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:19.068252 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:19.068790 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:19.068799 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:19.069240 (+ 441us) client_negotiation.cc:769] Sending connection context
0926 02:18:19.069521 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 02:18:19.070125 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":268,"threads_started":1}
I20260926 02:18:19.073055 15331 heartbeater.cc:347] Connected to a master server at 127.28.152.190:45409
I20260926 02:18:19.073444 15331 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:19.074414 15331 heartbeater.cc:511] Master 127.28.152.190:45409 requested a full tablet report, sending...
I20260926 02:18:19.076954 15141 ts_manager.cc:194] Registered new tserver with Master: 21b8fb95c5004e4da70c3228af24c4f2 (127.28.152.129:45485)
I20260926 02:18:19.079196 15141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:56525
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:18:19.112651 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45409
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:19.441636 15338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:19.441921 15338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:19.442058 15338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:19.449574 15338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:19.449704 15338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:19.449864 15338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:18:19.460081 15338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:19.461535 15338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:19.462900 15338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:19.476289 15343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:19.476302 15344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:19.476555 15346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:19.477612 15338 server_base.cc:1034] running on GCE node
I20260926 02:18:19.478202 15338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:19.479106 15338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:19.480296 15338 hybrid_clock.cc:648] HybridClock initialized: now 1790389099480249 us; error 53 us; skew 500 ppm
I20260926 02:18:19.480633 15338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:19 dist-test-slave-hllj krb5kdc[15096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:19.490033 15338 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:18:19.491644 15338 webserver.cc:460] Webserver started at http://127.28.152.130:46801/ using document root <none> and password file <none>
I20260926 02:18:19.492363 15338 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:19.492473 15338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:19.492782 15338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:19.496057 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "7a1a779b91a34753a41e404bc7e3d08c"
format_stamp: "Formatted at 2026-09-26 02:18:19 on dist-test-slave-hllj"
server_key: "d664af0193e3ce10f0f8997c3eae51a3"
server_key_iv: "6f498a29e027ed7575de5cdb5c4a7472"
server_key_version: "encryptionkey@0"
I20260926 02:18:19.496824 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "7a1a779b91a34753a41e404bc7e3d08c"
format_stamp: "Formatted at 2026-09-26 02:18:19 on dist-test-slave-hllj"
server_key: "d664af0193e3ce10f0f8997c3eae51a3"
server_key_iv: "6f498a29e027ed7575de5cdb5c4a7472"
server_key_version: "encryptionkey@0"
I20260926 02:18:19.502288 15338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:18:19.505999 15353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:19.507449 15338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:19.507644 15338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "7a1a779b91a34753a41e404bc7e3d08c"
format_stamp: "Formatted at 2026-09-26 02:18:19 on dist-test-slave-hllj"
server_key: "d664af0193e3ce10f0f8997c3eae51a3"
server_key_iv: "6f498a29e027ed7575de5cdb5c4a7472"
server_key_version: "encryptionkey@0"
I20260926 02:18:19.507850 15338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:19.522764 15338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:19.527328 15338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:19.527637 15338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:19.528664 15338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:19.530133 15338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:19.530215 15338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:19.530365 15338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:19.530433 15338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:19.562335 15338 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:34095
I20260926 02:18:19.562369 15465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:34095 every 8 connection(s)
I20260926 02:18:19.563710 15338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:18:19.566591 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 15338
I20260926 02:18:19.566784 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:18:19.567308 29282 external_mini_cluster.cc:1428] Setting key fc4e852bb9c9e43adad2b35614847b89
Sep 26 02:18:19 dist-test-slave-hllj krb5kdc[15096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:19.586768 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:19.566441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:32785 (local address 127.28.152.190:45409)
0926 02:18:19.566662 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:19.566668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:19.568138 (+ 1470us) server_negotiation.cc:407] Connection header received
0926 02:18:19.570281 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:19.570298 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:19.570390 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:19.570498 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:19.572447 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:19.573365 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:19.575085 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:19.575559 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:19.575680 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:19.579928 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:19.579935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:19.579939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:19.579983 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:19.582823 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:19.583998 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:19.584004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:19.584006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:19.584175 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:19.584761 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:19.584765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:19.584767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:19.585170 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:19.585310 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:19.586292 (+ 982us) server_negotiation.cc:299] Negotiation successful
0926 02:18:19.586516 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:18:19.588261 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:19.567051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45409 (local address 127.28.152.130:32785)
0926 02:18:19.567964 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:19.568022 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:19.569938 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:19.570824 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:19.570834 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:19.571533 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:19.572202 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:19.572223 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:19.573560 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:19.573564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:19.574909 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:19.574924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:19.575815 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:19.575823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:19.576109 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:19.576881 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:18:19.576907 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:19.579587 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:19.583054 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:19.583065 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:19.583086 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:19.583801 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:19.584308 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:19.584314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:19.584318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:19.584622 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:19.585303 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:19.585313 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:19.585841 (+ 528us) client_negotiation.cc:769] Sending connection context
0926 02:18:19.586170 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 02:18:19.586787 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":345,"threads_started":1}
I20260926 02:18:19.590014 15466 heartbeater.cc:347] Connected to a master server at 127.28.152.190:45409
I20260926 02:18:19.590395 15466 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:19.591238 15466 heartbeater.cc:511] Master 127.28.152.190:45409 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
I20260926 02:18:19.592865 15141 ts_manager.cc:194] Registered new tserver with Master: 7a1a779b91a34753a41e404bc7e3d08c (127.28.152.130:34095)
I20260926 02:18:19.593804 15141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:32785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:18:19.624333 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45409
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:19.958158 15471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:19.958473 15471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:19.958601 15471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:19.966153 15471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:19.966277 15471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:19.966441 15471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:18:19.976444 15471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:45409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:19.977836 15471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:19.979183 15471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:19.992558 15478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:19.992576 15480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:19.992658 15477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:19.993826 15471 server_base.cc:1034] running on GCE node
I20260926 02:18:19.994495 15471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:19.995558 15471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:19.996760 15471 hybrid_clock.cc:648] HybridClock initialized: now 1790389099996702 us; error 47 us; skew 500 ppm
I20260926 02:18:19.997092 15471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:19 dist-test-slave-hllj krb5kdc[15096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:20.000223 15471 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:18:20.001771 15471 webserver.cc:460] Webserver started at http://127.28.152.131:37669/ using document root <none> and password file <none>
I20260926 02:18:20.002473 15471 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:20.002583 15471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:20.002885 15471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:20.006084 15471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "dc5555882a594840bafb079dc095caf0"
format_stamp: "Formatted at 2026-09-26 02:18:19 on dist-test-slave-hllj"
server_key: "efe876d683598fbcf3a7c0fa912a4703"
server_key_iv: "50755cca26e2f7096a3a1fdf7a9e910b"
server_key_version: "encryptionkey@0"
I20260926 02:18:20.006861 15471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "dc5555882a594840bafb079dc095caf0"
format_stamp: "Formatted at 2026-09-26 02:18:19 on dist-test-slave-hllj"
server_key: "efe876d683598fbcf3a7c0fa912a4703"
server_key_iv: "50755cca26e2f7096a3a1fdf7a9e910b"
server_key_version: "encryptionkey@0"
I20260926 02:18:20.012238 15471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:18:20.015759 15487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:20.017194 15471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:18:20.017346 15471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "dc5555882a594840bafb079dc095caf0"
format_stamp: "Formatted at 2026-09-26 02:18:19 on dist-test-slave-hllj"
server_key: "efe876d683598fbcf3a7c0fa912a4703"
server_key_iv: "50755cca26e2f7096a3a1fdf7a9e910b"
server_key_version: "encryptionkey@0"
I20260926 02:18:20.017580 15471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:20.044875 15471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:20.050138 15471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:20.050485 15471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:20.051697 15471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:20.053397 15471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:20.053508 15471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:20.053633 15471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:20.053692 15471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:20.083506 15331 heartbeater.cc:503] Master 127.28.152.190:45409 was elected leader, sending a full tablet report...
I20260926 02:18:20.083719 15471 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:34453
I20260926 02:18:20.083758 15599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:34453 every 8 connection(s)
I20260926 02:18:20.085394 15471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:18:20.089077 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 15471
I20260926 02:18:20.089301 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:18:20.089872 29282 external_mini_cluster.cc:1428] Setting key c5c25cfca973a596d98dead0bb006d29
Sep 26 02:18:20 dist-test-slave-hllj krb5kdc[15096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:20.106552 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.088037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:44897 (local address 127.28.152.190:45409)
0926 02:18:20.088487 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:20.088493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:20.089827 (+ 1334us) server_negotiation.cc:407] Connection header received
0926 02:18:20.092205 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:20.092209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:20.092279 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:20.092367 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:20.094252 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.095106 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.096717 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.097158 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.097259 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.100661 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:20.100666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:20.100670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:20.100706 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:20.103255 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:20.104121 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:20.104127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:20.104130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:20.104292 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:20.104761 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:20.104766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:20.104768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:20.105118 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:20.105297 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:20.106088 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 02:18:20.106262 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":215,"threads_started":1}
I20260926 02:18:20.107520 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.088694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45409 (local address 127.28.152.131:44897)
0926 02:18:20.089665 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:20.089705 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:20.091895 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:20.092670 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:20.092683 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:20.093388 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:20.094082 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.094103 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.095263 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.095270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.096586 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.096598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.097335 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.097340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.097558 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.098174 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:18:20.098196 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:20.100413 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:20.103411 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:20.103420 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:20.103442 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:20.104011 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:20.104399 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:20.104403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:20.104406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:20.104655 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:20.105281 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:20.105290 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:20.105731 (+ 441us) client_negotiation.cc:769] Sending connection context
0926 02:18:20.105992 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 02:18:20.106399 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":405,"threads_started":1}
I20260926 02:18:20.108902 15600 heartbeater.cc:347] Connected to a master server at 127.28.152.190:45409
I20260926 02:18:20.109262 15600 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:20.110021 15600 heartbeater.cc:511] Master 127.28.152.190:45409 requested a full tablet report, sending...
I20260926 02:18:20.111459 15141 ts_manager.cc:194] Registered new tserver with Master: dc5555882a594840bafb079dc095caf0 (127.28.152.131:34453)
I20260926 02:18:20.112267 15141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:44897
I20260926 02:18:20.117762 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:18:20.134801 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.121740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.28.152.190:45409)
0926 02:18:20.121915 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:20.121921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:20.122484 (+ 563us) server_negotiation.cc:407] Connection header received
0926 02:18:20.122618 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:20.122622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:20.122686 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:20.122766 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:20.124210 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.124924 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.126334 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.126734 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.126828 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.129534 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:20.129540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:20.129543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:20.129581 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:20.132113 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:20.132940 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:20.132944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:20.132946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:20.133086 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:20.133500 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:20.133503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:20.133505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:20.133839 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:20.134046 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:20.134370 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 02:18:20.134564 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:18:20.138717 15141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:18:20.145735 15141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:18:20.184549 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.158339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:34095 (local address 127.0.0.1:35064)
0926 02:18:20.175371 (+ 17032us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:20.175389 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:20.175641 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:20.176993 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:20.176998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:20.177033 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:20.177551 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.177561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.179232 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.179237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.181596 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.181613 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.183443 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.183450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.183667 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.183728 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 02:18:20.183868 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 02:18:20.184097 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16909,"mutex_wait_us":291,"thread_start_us":9854,"threads_started":1}
I20260926 02:18:20.184913 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.158334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45485 (local address 127.0.0.1:38248)
0926 02:18:20.168929 (+ 10595us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:20.168990 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:20.169275 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:20.176870 (+ 7595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:20.176878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:20.176911 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:20.177487 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.177505 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.180293 (+ 2788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.180299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.182281 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.182292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.184022 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.184029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.184194 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.184227 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:18:20.184296 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 02:18:20.184419 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10231,"mutex_wait_us":52,"thread_start_us":16726,"threads_started":1}
I20260926 02:18:20.185382 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.158369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.28.152.130:34095)
0926 02:18:20.174876 (+ 16507us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:20.174886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:20.175514 (+ 628us) server_negotiation.cc:407] Connection header received
0926 02:18:20.176379 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:20.176386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:20.176575 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:20.176772 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:20.177719 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.179072 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.181786 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.183217 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.183409 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.183607 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:20.184835 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0926 02:18:20.185030 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16334,"thread_start_us":16220,"threads_started":1}
I20260926 02:18:20.185483 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.158978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.28.152.129:45485)
0926 02:18:20.174553 (+ 15575us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:20.174564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:20.174606 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:18:20.176243 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:20.176255 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:20.176577 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:20.177756 (+ 1179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:20.178014 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.180153 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.182443 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.183789 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.184693 (+ 904us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.184886 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:20.184980 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 02:18:20.185163 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15372,"thread_start_us":15178,"threads_started":1}
I20260926 02:18:20.186126 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.175069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:34453 (local address 127.0.0.1:40536)
0926 02:18:20.176986 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:20.177003 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:20.177148 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:20.178044 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:20.178049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:20.178077 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:20.178429 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.178441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.180447 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.180452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.182265 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.182281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.185289 (+ 3008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.185296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.185457 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.185492 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:18:20.185576 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 02:18:20.185705 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1777,"mutex_wait_us":1,"thread_start_us":1625,"threads_started":1}
I20260926 02:18:20.187153 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.169946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.28.152.131:34453)
0926 02:18:20.176226 (+ 6280us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:20.176234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:20.177112 (+ 878us) server_negotiation.cc:407] Connection header received
0926 02:18:20.177290 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:20.177298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:20.177567 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:20.177753 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:20.178649 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.180303 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.183044 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.184877 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.185060 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.185237 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:20.185875 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 02:18:20.186061 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6109,"thread_start_us":5946,"threads_started":1}
I20260926 02:18:20.189810 15401 tablet_service.cc:1467] Processing CreateTablet for tablet 4e74803fe0d845fbb9fb34c0341930e0 (DEFAULT_TABLE table=test-table [id=7906911b61e64d4c98de8fb72bd4a138]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:20.189811 15266 tablet_service.cc:1467] Processing CreateTablet for tablet 4e74803fe0d845fbb9fb34c0341930e0 (DEFAULT_TABLE table=test-table [id=7906911b61e64d4c98de8fb72bd4a138]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:20.191129 15401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e74803fe0d845fbb9fb34c0341930e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:20.191222 15266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e74803fe0d845fbb9fb34c0341930e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:20.190862 15535 tablet_service.cc:1467] Processing CreateTablet for tablet 4e74803fe0d845fbb9fb34c0341930e0 (DEFAULT_TABLE table=test-table [id=7906911b61e64d4c98de8fb72bd4a138]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:20.192241 15535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e74803fe0d845fbb9fb34c0341930e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:20.203639 15619 tablet_bootstrap.cc:492] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: Bootstrap starting.
I20260926 02:18:20.206347 15621 tablet_bootstrap.cc:492] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: Bootstrap starting.
I20260926 02:18:20.207967 15620 tablet_bootstrap.cc:492] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: Bootstrap starting.
I20260926 02:18:20.209117 15619 tablet_bootstrap.cc:654] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:20.210147 15621 tablet_bootstrap.cc:654] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:20.210556 15619 log.cc:824] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:20.211066 15620 tablet_bootstrap.cc:654] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:20.211599 15621 log.cc:824] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:20.212126 15620 log.cc:824] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:20.214762 15621 tablet_bootstrap.cc:492] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: No bootstrap required, opened a new log
I20260926 02:18:20.214828 15620 tablet_bootstrap.cc:492] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: No bootstrap required, opened a new log
I20260926 02:18:20.215057 15621 ts_tablet_manager.cc:1397] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 02:18:20.215072 15620 ts_tablet_manager.cc:1397] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 02:18:20.217118 15619 tablet_bootstrap.cc:492] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: No bootstrap required, opened a new log
I20260926 02:18:20.217461 15619 ts_tablet_manager.cc:1397] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260926 02:18:20.223052 15621 raft_consensus.cc:348] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1a779b91a34753a41e404bc7e3d08c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34095 } } peers { permanent_uuid: "21b8fb95c5004e4da70c3228af24c4f2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45485 } } peers { permanent_uuid: "dc5555882a594840bafb079dc095caf0" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 34453 } }
I20260926 02:18:20.223446 15621 raft_consensus.cc:374] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:20.223562 15621 raft_consensus.cc:729] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5555882a594840bafb079dc095caf0, State: Initialized, Role: FOLLOWER
I20260926 02:18:20.224223 15621 consensus_queue.cc:260] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1a779b91a34753a41e404bc7e3d08c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34095 } } peers { permanent_uuid: "21b8fb95c5004e4da70c3228af24c4f2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45485 } } peers { permanent_uuid: "dc5555882a594840bafb079dc095caf0" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 34453 } }
I20260926 02:18:20.225332 15600 heartbeater.cc:503] Master 127.28.152.190:45409 was elected leader, sending a full tablet report...
I20260926 02:18:20.225512 15620 raft_consensus.cc:348] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1a779b91a34753a41e404bc7e3d08c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34095 } } peers { permanent_uuid: "21b8fb95c5004e4da70c3228af24c4f2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45485 } } peers { permanent_uuid: "dc5555882a594840bafb079dc095caf0" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 34453 } }
I20260926 02:18:20.225950 15620 raft_consensus.cc:374] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:20.226037 15620 raft_consensus.cc:729] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b8fb95c5004e4da70c3228af24c4f2, State: Initialized, Role: FOLLOWER
I20260926 02:18:20.226071 15621 ts_tablet_manager.cc:1428] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: Time spent starting tablet: real 0.011s user 0.001s sys 0.009s
I20260926 02:18:20.226734 15620 consensus_queue.cc:260] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1a779b91a34753a41e404bc7e3d08c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34095 } } peers { permanent_uuid: "21b8fb95c5004e4da70c3228af24c4f2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45485 } } peers { permanent_uuid: "dc5555882a594840bafb079dc095caf0" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 34453 } }
I20260926 02:18:20.227793 15619 raft_consensus.cc:348] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1a779b91a34753a41e404bc7e3d08c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34095 } } peers { permanent_uuid: "21b8fb95c5004e4da70c3228af24c4f2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45485 } } peers { permanent_uuid: "dc5555882a594840bafb079dc095caf0" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 34453 } }
I20260926 02:18:20.228305 15619 raft_consensus.cc:374] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:20.228456 15619 raft_consensus.cc:729] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1a779b91a34753a41e404bc7e3d08c, State: Initialized, Role: FOLLOWER
I20260926 02:18:20.228673 15620 ts_tablet_manager.cc:1428] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260926 02:18:20.229224 15619 consensus_queue.cc:260] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1a779b91a34753a41e404bc7e3d08c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34095 } } peers { permanent_uuid: "21b8fb95c5004e4da70c3228af24c4f2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45485 } } peers { permanent_uuid: "dc5555882a594840bafb079dc095caf0" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 34453 } }
I20260926 02:18:20.230537 15466 heartbeater.cc:503] Master 127.28.152.190:45409 was elected leader, sending a full tablet report...
I20260926 02:18:20.231099 15619 ts_tablet_manager.cc:1428] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260926 02:18:20.301301 15333 tablet.cc:2369] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:18:20.315661 15467 tablet.cc:2369] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:18:20.337033 15601 tablet.cc:2369] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:18:20.535251 15625 raft_consensus.cc:482] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:18:20.535552 15625 raft_consensus.cc:504] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1a779b91a34753a41e404bc7e3d08c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34095 } } peers { permanent_uuid: "21b8fb95c5004e4da70c3228af24c4f2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45485 } } peers { permanent_uuid: "dc5555882a594840bafb079dc095caf0" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 34453 } }
I20260926 02:18:20.537298 15625 leader_election.cc:288] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a1a779b91a34753a41e404bc7e3d08c (127.28.152.130:34095), 21b8fb95c5004e4da70c3228af24c4f2 (127.28.152.129:45485)
I20260926 02:18:20.543728 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.537759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:34095 (local address 127.28.152.131:41667)
0926 02:18:20.537994 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:20.538013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:20.538147 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:20.538476 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:20.538482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:20.538509 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:20.538689 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.538701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.539985 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.539991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.541631 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.541644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.542969 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.542975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.543164 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.543200 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:18:20.543300 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 02:18:20.543449 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 02:18:20.543723 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.537794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:41667 (local address 127.28.152.130:34095)
0926 02:18:20.538162 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:20.538167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:20.538189 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:18:20.538262 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:20.538268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:20.538332 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:20.538405 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:20.538862 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.539841 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.541803 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.542811 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.542930 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.542985 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:20.543322 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 02:18:20.543425 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260926 02:18:20.544317 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.537906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:35719 (local address 127.28.152.129:45485)
0926 02:18:20.538140 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:20.538147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:20.538686 (+ 539us) server_negotiation.cc:407] Connection header received
0926 02:18:20.538826 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:20.538832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:20.538894 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:20.538966 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:20.539563 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.540514 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.542413 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.543403 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.543548 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.543617 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:20.543986 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 02:18:20.544140 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:18:20.545001 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.537791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45485 (local address 127.28.152.131:35719)
0926 02:18:20.538572 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:20.538587 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:20.538706 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:20.539016 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:20.539021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:20.539056 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:20.539401 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.539412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.540665 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.540669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.542243 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:20.542257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.543622 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.543628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:20.543792 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.543824 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:18:20.544490 (+ 666us) client_negotiation.cc:240] Negotiation successful
0926 02:18:20.544632 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":1,"thread_start_us":467,"threads_started":1}
I20260926 02:18:20.544963 15421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e74803fe0d845fbb9fb34c0341930e0" candidate_uuid: "dc5555882a594840bafb079dc095caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a1a779b91a34753a41e404bc7e3d08c" is_pre_election: true
I20260926 02:18:20.545570 15421 raft_consensus.cc:2454] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc5555882a594840bafb079dc095caf0 in term 0.
I20260926 02:18:20.546130 15286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e74803fe0d845fbb9fb34c0341930e0" candidate_uuid: "dc5555882a594840bafb079dc095caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b8fb95c5004e4da70c3228af24c4f2" is_pre_election: true
I20260926 02:18:20.546373 15489 leader_election.cc:302] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a1a779b91a34753a41e404bc7e3d08c, dc5555882a594840bafb079dc095caf0; no voters:
I20260926 02:18:20.546710 15286 raft_consensus.cc:2454] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc5555882a594840bafb079dc095caf0 in term 0.
I20260926 02:18:20.546826 15625 raft_consensus.cc:2781] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:18:20.546999 15625 raft_consensus.cc:482] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:18:20.547092 15625 raft_consensus.cc:3037] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:20.549368 15625 raft_consensus.cc:504] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1a779b91a34753a41e404bc7e3d08c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34095 } } peers { permanent_uuid: "21b8fb95c5004e4da70c3228af24c4f2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45485 } } peers { permanent_uuid: "dc5555882a594840bafb079dc095caf0" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 34453 } }
I20260926 02:18:20.549958 15625 leader_election.cc:288] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [CANDIDATE]: Term 1 election: Requested vote from peers 7a1a779b91a34753a41e404bc7e3d08c (127.28.152.130:34095), 21b8fb95c5004e4da70c3228af24c4f2 (127.28.152.129:45485)
I20260926 02:18:20.550495 15421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e74803fe0d845fbb9fb34c0341930e0" candidate_uuid: "dc5555882a594840bafb079dc095caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a1a779b91a34753a41e404bc7e3d08c"
I20260926 02:18:20.550495 15286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e74803fe0d845fbb9fb34c0341930e0" candidate_uuid: "dc5555882a594840bafb079dc095caf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b8fb95c5004e4da70c3228af24c4f2"
I20260926 02:18:20.550714 15286 raft_consensus.cc:3037] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:20.550710 15421 raft_consensus.cc:3037] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:20.553876 15286 raft_consensus.cc:2454] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc5555882a594840bafb079dc095caf0 in term 1.
I20260926 02:18:20.553876 15421 raft_consensus.cc:2454] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc5555882a594840bafb079dc095caf0 in term 1.
I20260926 02:18:20.554324 15489 leader_election.cc:302] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a1a779b91a34753a41e404bc7e3d08c, dc5555882a594840bafb079dc095caf0; no voters:
I20260926 02:18:20.554632 15625 raft_consensus.cc:2781] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:20.555083 15625 raft_consensus.cc:686] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 1 LEADER]: Becoming Leader. State: Replica: dc5555882a594840bafb079dc095caf0, State: Running, Role: LEADER
I20260926 02:18:20.555552 15625 consensus_queue.cc:237] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1a779b91a34753a41e404bc7e3d08c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34095 } } peers { permanent_uuid: "21b8fb95c5004e4da70c3228af24c4f2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45485 } } peers { permanent_uuid: "dc5555882a594840bafb079dc095caf0" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 34453 } }
I20260926 02:18:20.560336 15139 catalog_manager.cc:5495] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc5555882a594840bafb079dc095caf0 (127.28.152.131). New cstate: current_term: 1 leader_uuid: "dc5555882a594840bafb079dc095caf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a1a779b91a34753a41e404bc7e3d08c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21b8fb95c5004e4da70c3228af24c4f2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc5555882a594840bafb079dc095caf0" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 34453 } health_report { overall_health: HEALTHY } } }
I20260926 02:18:20.655164 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.644022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.28.152.131:34453)
0926 02:18:20.644206 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:20.644211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:20.649747 (+ 5536us) server_negotiation.cc:407] Connection header received
0926 02:18:20.649921 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:20.649925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:20.650022 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:20.650174 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:18:20.650664 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.651461 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.652961 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.653452 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.653592 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.654095 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:18:20.654653 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:18:20.654794 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:20.654909 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 02:18:20.654953 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 02:18:20.663625 15286 raft_consensus.cc:1261] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2 [term 1 FOLLOWER]: Refusing update from remote peer dc5555882a594840bafb079dc095caf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:20.663625 15421 raft_consensus.cc:1261] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c [term 1 FOLLOWER]: Refusing update from remote peer dc5555882a594840bafb079dc095caf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:20.664665 15629 consensus_queue.cc:1035] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b8fb95c5004e4da70c3228af24c4f2" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:20.664994 15625 consensus_queue.cc:1035] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1a779b91a34753a41e404bc7e3d08c" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:20.679842 15635 mvcc.cc:205] Tried to move back new op lower bound from 7333433756308328448 to 7333433755879309312. Current Snapshot: MvccSnapshot[applied={T|T < 7333433756308328448}]
I20260926 02:18:20.691455 15139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:20.691697 15139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:20.695152 15139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f37f6f2d0fb494f9353d1dcf9819dfc: Sending DeleteTablet for 3 replicas of tablet 4e74803fe0d845fbb9fb34c0341930e0
I20260926 02:18:20.696456 15266 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e74803fe0d845fbb9fb34c0341930e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:20 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:38248
I20260926 02:18:20.696514 15401 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e74803fe0d845fbb9fb34c0341930e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:20 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:35064
I20260926 02:18:20.696679 15535 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e74803fe0d845fbb9fb34c0341930e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:20 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:40536
I20260926 02:18:20.697588 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 15203
I20260926 02:18:20.708689 15645 tablet_replica.cc:330] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: stopping tablet replica
I20260926 02:18:20.709450 15645 raft_consensus.cc:2229] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:18:20.710216 15644 tablet_replica.cc:330] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: stopping tablet replica
I20260926 02:18:20.710695 15644 raft_consensus.cc:2229] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:20.711266 15644 raft_consensus.cc:2258] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:20.712258 15645 raft_consensus.cc:2258] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:20.713099 15643 tablet_replica.cc:330] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: stopping tablet replica
I20260926 02:18:20.713603 15643 raft_consensus.cc:2229] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:20.714197 15643 raft_consensus.cc:2258] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:20.716701 15645 ts_tablet_manager.cc:1905] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:20.719022 15644 ts_tablet_manager.cc:1905] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:20.722098 15643 ts_tablet_manager.cc:1905] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:20.724529 15645 ts_tablet_manager.cc:1918] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:20.724715 15645 log.cc:1196] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/4e74803fe0d845fbb9fb34c0341930e0
I20260926 02:18:20.725291 15645 ts_tablet_manager.cc:1939] T 4e74803fe0d845fbb9fb34c0341930e0 P dc5555882a594840bafb079dc095caf0: Deleting consensus metadata
I20260926 02:18:20.727504 15126 catalog_manager.cc:4841] TS dc5555882a594840bafb079dc095caf0 (127.28.152.131:34453): tablet 4e74803fe0d845fbb9fb34c0341930e0 (table test-table [id=7906911b61e64d4c98de8fb72bd4a138]) successfully deleted
I20260926 02:18:20.728652 15643 ts_tablet_manager.cc:1918] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:20.728814 15643 log.cc:1196] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/4e74803fe0d845fbb9fb34c0341930e0
Sep 26 02:18:20 dist-test-slave-hllj krb5kdc[15096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:18:20.729394 15643 ts_tablet_manager.cc:1939] T 4e74803fe0d845fbb9fb34c0341930e0 P 21b8fb95c5004e4da70c3228af24c4f2: Deleting consensus metadata
I20260926 02:18:20.732839 15125 catalog_manager.cc:4841] TS 21b8fb95c5004e4da70c3228af24c4f2 (127.28.152.129:45485): tablet 4e74803fe0d845fbb9fb34c0341930e0 (table test-table [id=7906911b61e64d4c98de8fb72bd4a138]) successfully deleted
I20260926 02:18:20.735167 15644 ts_tablet_manager.cc:1918] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:20.735371 15644 log.cc:1196] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/4e74803fe0d845fbb9fb34c0341930e0
I20260926 02:18:20.735813 15644 ts_tablet_manager.cc:1939] T 4e74803fe0d845fbb9fb34c0341930e0 P 7a1a779b91a34753a41e404bc7e3d08c: Deleting consensus metadata
I20260926 02:18:20.737942 15126 catalog_manager.cc:4841] TS 7a1a779b91a34753a41e404bc7e3d08c (127.28.152.130:34095): tablet 4e74803fe0d845fbb9fb34c0341930e0 (table test-table [id=7906911b61e64d4c98de8fb72bd4a138]) successfully deleted
I20260926 02:18:20.738855 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.698419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.28.152.129:45485)
0926 02:18:20.698608 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:20.698615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:20.717743 (+ 19128us) server_negotiation.cc:407] Connection header received
0926 02:18:20.717966 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:20.717972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:20.718064 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:20.718168 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:20.720137 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.721314 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.723565 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.724245 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.724371 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.730375 (+ 6004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:20.730391 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:20.730411 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:20.730465 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:20.734472 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:20.735549 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:20.735560 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:20.735564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:20.735744 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:20.736242 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:20.736247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:20.736251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:20.736746 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:20.738243 (+ 1497us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:20.738390 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 02:18:20.738575 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:18:20.739524 15326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:20.828837 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 15203
I20260926 02:18:20.843603 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 15338
Sep 26 02:18:20 dist-test-slave-hllj krb5kdc[15096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:18:20.862013 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.844555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.28.152.130:34095)
0926 02:18:20.844718 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:20.844722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:20.844741 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:18:20.844893 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:20.844903 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:20.844982 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:20.845062 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:20.846874 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.847658 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.849785 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.850213 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.850337 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.855891 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:20.855916 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:20.855934 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:20.855976 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:20.858841 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:20.859788 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:20.859797 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:20.859799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:20.859970 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:20.860387 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:20.860391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:20.860393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:20.860796 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:20.861047 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:20.861503 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 02:18:20.861693 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:18:20.862747 15461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:20.952438 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 15338
I20260926 02:18:20.967339 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 15471
Sep 26 02:18:20 dist-test-slave-hllj krb5kdc[15096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:18:20.987020 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:20.968208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.28.152.131:34453)
0926 02:18:20.968404 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:20.968411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:20.968637 (+ 226us) server_negotiation.cc:407] Connection header received
0926 02:18:20.968845 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:20.968850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:20.968932 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:20.969011 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:20.970937 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.971844 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.973868 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:20.974343 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:20.974460 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:20.980584 (+ 6124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:20.980598 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:20.980615 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:20.980661 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:20.983619 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:20.984564 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:20.984571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:20.984573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:20.984745 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:20.985420 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:20.985425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:20.985428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:20.985807 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:20.986085 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:20.986476 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 02:18:20.986669 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 02:18:20.987722 15595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:21.107969 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 15471
I20260926 02:18:21.124610 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 15109
I20260926 02:18:21.125803 15171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:21.234704 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 15109
2026-09-26T02:18:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4516 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T02:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:21Z Disabled control of system clock
I20260926 02:18:21.305995 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:46615
--webserver_interface=127.28.152.190
--webserver_port=0
--builtin_ntp_servers=127.28.152.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:46615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:18:21.635692 15654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:21.636080 15654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:21.636205 15654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:21.643728 15654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:21.643854 15654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:21.643916 15654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:21.643993 15654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:21.644049 15654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:21.653554 15654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:46615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:21.654910 15654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:21.656806 15654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:21.668607 15662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:21.668726 15660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:21.668919 15659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:21.670107 15654 server_base.cc:1034] running on GCE node
I20260926 02:18:21.670944 15654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:21.672400 15654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:21.673647 15654 hybrid_clock.cc:648] HybridClock initialized: now 1790389101673577 us; error 46 us; skew 500 ppm
I20260926 02:18:21.674023 15654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:21.676273 15654 webserver.cc:460] Webserver started at http://127.28.152.190:43775/ using document root <none> and password file <none>
I20260926 02:18:21.676982 15654 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:21.677083 15654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:21.677412 15654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:21.680580 15654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "2a8c2af3e82747cba079e7773316ea88"
format_stamp: "Formatted at 2026-09-26 02:18:21 on dist-test-slave-hllj"
server_key: "35767f6e995aeab46b02a1ec4721c362"
server_key_iv: "6e22087bdac2a1fa6b186a43be5fdc24"
server_key_version: "encryptionkey@0"
I20260926 02:18:21.681483 15654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "2a8c2af3e82747cba079e7773316ea88"
format_stamp: "Formatted at 2026-09-26 02:18:21 on dist-test-slave-hllj"
server_key: "35767f6e995aeab46b02a1ec4721c362"
server_key_iv: "6e22087bdac2a1fa6b186a43be5fdc24"
server_key_version: "encryptionkey@0"
I20260926 02:18:21.686852 15654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:18:21.690389 15668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:21.691761 15654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:21.691944 15654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "2a8c2af3e82747cba079e7773316ea88"
format_stamp: "Formatted at 2026-09-26 02:18:21 on dist-test-slave-hllj"
server_key: "35767f6e995aeab46b02a1ec4721c362"
server_key_iv: "6e22087bdac2a1fa6b186a43be5fdc24"
server_key_version: "encryptionkey@0"
I20260926 02:18:21.692152 15654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:21.718223 15654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:21.719125 15654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:21.719455 15654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:21.736995 15654 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:46615
I20260926 02:18:21.737020 15719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:46615 every 8 connection(s)
I20260926 02:18:21.738453 15654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:21.739997 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 15654
I20260926 02:18:21.740151 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:21.740573 29282 external_mini_cluster.cc:1428] Setting key 1f5c5544b370c09e41288bc66d0be948
I20260926 02:18:21.742913 15720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:21.754568 15720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88: Bootstrap starting.
I20260926 02:18:21.754683 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:21.742463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.28.152.190:46615)
0926 02:18:21.743575 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:21.743587 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:21.743634 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:18:21.745419 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:21.745444 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:21.745991 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:21.746422 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:21.747039 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:21.748628 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:21.750112 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:21.750825 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:21.751006 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:21.752194 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:21.752205 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:21.752222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:21.752820 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:21.752906 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:21.752933 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:21.753035 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:21.753314 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 02:18:21.753660 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":340,"threads_started":1}
I20260926 02:18:21.758055 15720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:21.759088 15720 log.cc:824] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:21.762068 15720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88: No bootstrap required, opened a new log
I20260926 02:18:21.770007 15720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8c2af3e82747cba079e7773316ea88" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46615 } }
I20260926 02:18:21.770462 15720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:21.770572 15720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8c2af3e82747cba079e7773316ea88, State: Initialized, Role: FOLLOWER
I20260926 02:18:21.771203 15720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8c2af3e82747cba079e7773316ea88" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46615 } }
I20260926 02:18:21.771422 15720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:21.771523 15720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:21.771660 15720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:21.774174 15720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8c2af3e82747cba079e7773316ea88" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46615 } }
I20260926 02:18:21.774721 15720 leader_election.cc:302] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a8c2af3e82747cba079e7773316ea88; no voters:
I20260926 02:18:21.775266 15720 leader_election.cc:288] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:21.775444 15725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:21.776129 15725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [term 1 LEADER]: Becoming Leader. State: Replica: 2a8c2af3e82747cba079e7773316ea88, State: Running, Role: LEADER
I20260926 02:18:21.776647 15725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8c2af3e82747cba079e7773316ea88" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46615 } }
I20260926 02:18:21.777499 15720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:21.779436 15727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a8c2af3e82747cba079e7773316ea88. Latest consensus state: current_term: 1 leader_uuid: "2a8c2af3e82747cba079e7773316ea88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8c2af3e82747cba079e7773316ea88" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46615 } } }
I20260926 02:18:21.779820 15727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:21.780146 15726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a8c2af3e82747cba079e7773316ea88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8c2af3e82747cba079e7773316ea88" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 46615 } } }
I20260926 02:18:21.780465 15726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:21.782240 15731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:21.788268 15731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:21.798135 15731 catalog_manager.cc:1347] Generated new cluster ID: 3ca83cabb9d447a3b40a76045c1a4357
I20260926 02:18:21.798259 15731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:21.825759 15731 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:21.827176 15731 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:21.836882 15731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88: Generated new TSK 0
I20260926 02:18:21.837827 15731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:21.843036 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46615
--builtin_ntp_servers=127.28.152.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:18:22.199028 15744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:22.199338 15744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:22.199460 15744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:22.206607 15744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:22.206707 15744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:22.206822 15744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:18:22.215922 15744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:22.217218 15744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:22.218940 15744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:22.231827 15749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:22.231798 15750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:22.231922 15752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:22.232904 15744 server_base.cc:1034] running on GCE node
I20260926 02:18:22.233553 15744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:22.234543 15744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:22.235743 15744 hybrid_clock.cc:648] HybridClock initialized: now 1790389102235694 us; error 35 us; skew 500 ppm
I20260926 02:18:22.236063 15744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:22.238382 15744 webserver.cc:460] Webserver started at http://127.28.152.129:46881/ using document root <none> and password file <none>
I20260926 02:18:22.239102 15744 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:22.239195 15744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:22.239500 15744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:22.242478 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "3bab7c2a58f3440fa245b75adeca92bf"
format_stamp: "Formatted at 2026-09-26 02:18:22 on dist-test-slave-hllj"
server_key: "5e2810fa2d0225b30fece12278470643"
server_key_iv: "3f31fa9be2300b6f13578df81f092012"
server_key_version: "encryptionkey@0"
I20260926 02:18:22.243271 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "3bab7c2a58f3440fa245b75adeca92bf"
format_stamp: "Formatted at 2026-09-26 02:18:22 on dist-test-slave-hllj"
server_key: "5e2810fa2d0225b30fece12278470643"
server_key_iv: "3f31fa9be2300b6f13578df81f092012"
server_key_version: "encryptionkey@0"
I20260926 02:18:22.248597 15744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 02:18:22.252133 15758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:22.253542 15744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:18:22.253692 15744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "3bab7c2a58f3440fa245b75adeca92bf"
format_stamp: "Formatted at 2026-09-26 02:18:22 on dist-test-slave-hllj"
server_key: "5e2810fa2d0225b30fece12278470643"
server_key_iv: "3f31fa9be2300b6f13578df81f092012"
server_key_version: "encryptionkey@0"
I20260926 02:18:22.253898 15744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:22.280200 15744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:22.281075 15744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:22.281399 15744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:22.282402 15744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:22.283885 15744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:22.283980 15744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:22.284088 15744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:22.284158 15744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:22.314841 15744 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:37091
I20260926 02:18:22.314904 15870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:37091 every 8 connection(s)
I20260926 02:18:22.316161 15744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:18:22.321873 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 15744
I20260926 02:18:22.322052 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:18:22.322592 29282 external_mini_cluster.cc:1428] Setting key 74023ad007280f9925c6cb08526d2c69
I20260926 02:18:22.325842 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46615
--builtin_ntp_servers=127.28.152.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:18:22.331212 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:22.318673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:53945 (local address 127.28.152.190:46615)
0926 02:18:22.319151 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:22.319157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:22.320348 (+ 1191us) server_negotiation.cc:407] Connection header received
0926 02:18:22.322627 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:22.322632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:22.322724 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:22.322826 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:22.324479 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:22.325505 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:22.327802 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:22.328378 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:22.328493 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:22.330295 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:22.330300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:22.330304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:22.330327 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:22.330385 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:22.330399 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:22.330477 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:22.330790 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 02:18:22.330890 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":219,"threads_started":1}
I20260926 02:18:22.332656 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:22.319301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:46615 (local address 127.28.152.129:53945)
0926 02:18:22.320193 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:22.320230 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:22.322152 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:22.323273 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:22.323286 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:22.323445 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:22.324262 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:22.324290 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:22.325690 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:22.325695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:22.327628 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:22.327645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:22.328788 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:22.328795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:22.329085 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:22.329881 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:18:22.329884 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:22.329940 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:18:22.330050 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:22.330531 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:22.330545 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:22.330555 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:18:22.330781 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 02:18:22.331246 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":249,"threads_started":1}
I20260926 02:18:22.334991 15871 heartbeater.cc:347] Connected to a master server at 127.28.152.190:46615
I20260926 02:18:22.335367 15871 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:22.336402 15871 heartbeater.cc:511] Master 127.28.152.190:46615 requested a full tablet report, sending...
I20260926 02:18:22.339517 15685 ts_manager.cc:194] Registered new tserver with Master: 3bab7c2a58f3440fa245b75adeca92bf (127.28.152.129:37091)
I20260926 02:18:22.341486 15685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.129:53945
W20260926 02:18:22.661391 15875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:22.661707 15875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:22.661839 15875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:22.669780 15875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:22.669914 15875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:22.670061 15875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:18:22.679821 15875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:22.681231 15875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:22.683087 15875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:22.697109 15881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:22.697224 15880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:22.697306 15883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:22.697832 15875 server_base.cc:1034] running on GCE node
I20260926 02:18:22.698580 15875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:22.699601 15875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:22.700830 15875 hybrid_clock.cc:648] HybridClock initialized: now 1790389102700774 us; error 49 us; skew 500 ppm
I20260926 02:18:22.701187 15875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:22.703636 15875 webserver.cc:460] Webserver started at http://127.28.152.130:41669/ using document root <none> and password file <none>
I20260926 02:18:22.704375 15875 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:22.704474 15875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:22.704788 15875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:22.707823 15875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "682f0e66099c4305b0cd7ec38d9d60c7"
format_stamp: "Formatted at 2026-09-26 02:18:22 on dist-test-slave-hllj"
server_key: "0bc6ad02c46823524a009a607210b94b"
server_key_iv: "b67127efc68f8ffe94489fa0f641385f"
server_key_version: "encryptionkey@0"
I20260926 02:18:22.708742 15875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "682f0e66099c4305b0cd7ec38d9d60c7"
format_stamp: "Formatted at 2026-09-26 02:18:22 on dist-test-slave-hllj"
server_key: "0bc6ad02c46823524a009a607210b94b"
server_key_iv: "b67127efc68f8ffe94489fa0f641385f"
server_key_version: "encryptionkey@0"
I20260926 02:18:22.720084 15875 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.010s sys 0.003s
I20260926 02:18:22.724073 15889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:22.725606 15875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:18:22.725778 15875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "682f0e66099c4305b0cd7ec38d9d60c7"
format_stamp: "Formatted at 2026-09-26 02:18:22 on dist-test-slave-hllj"
server_key: "0bc6ad02c46823524a009a607210b94b"
server_key_iv: "b67127efc68f8ffe94489fa0f641385f"
server_key_version: "encryptionkey@0"
I20260926 02:18:22.726003 15875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:22.743543 15875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:22.744463 15875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:22.744772 15875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:22.745872 15875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:22.747442 15875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:22.747548 15875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:22.747665 15875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:22.747735 15875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:22.778970 15875 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:46661
I20260926 02:18:22.779036 16001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:46661 every 8 connection(s)
I20260926 02:18:22.780285 15875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:18:22.790299 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 15875
I20260926 02:18:22.790473 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:18:22.791056 29282 external_mini_cluster.cc:1428] Setting key 21ec8728ee420978602ab04a583a9361
I20260926 02:18:22.793684 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:22.782976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:43021 (local address 127.28.152.190:46615)
0926 02:18:22.783165 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:22.783170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:22.784808 (+ 1638us) server_negotiation.cc:407] Connection header received
0926 02:18:22.786680 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:22.786686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:22.786772 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:22.786860 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:22.788042 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:22.788772 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:22.790542 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:22.791192 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:22.791345 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:22.792838 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:22.792844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:22.792848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:22.792870 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:22.792905 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:22.792919 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:22.792986 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:22.793275 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 02:18:22.793379 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:18:22.794059 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46615
--builtin_ntp_servers=127.28.152.148:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:18:22.794957 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:22.783557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:46615 (local address 127.28.152.130:43021)
0926 02:18:22.784564 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:22.784604 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:22.786367 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:22.787131 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:22.787143 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:22.787256 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:22.787860 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:22.787882 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:22.788934 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:22.788939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:22.790342 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:22.790357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:22.791464 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:22.791470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:22.791774 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:22.792502 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:18:22.792504 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:22.792548 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:18:22.792635 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:22.793009 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:22.793020 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:22.793029 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 02:18:22.793245 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 02:18:22.793734 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":277,"threads_started":1}
I20260926 02:18:22.796456 16003 heartbeater.cc:347] Connected to a master server at 127.28.152.190:46615
I20260926 02:18:22.796808 16003 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:22.797808 16003 heartbeater.cc:511] Master 127.28.152.190:46615 requested a full tablet report, sending...
I20260926 02:18:22.799497 15685 ts_manager.cc:194] Registered new tserver with Master: 682f0e66099c4305b0cd7ec38d9d60c7 (127.28.152.130:46661)
I20260926 02:18:22.800499 15685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.130:43021
W20260926 02:18:23.130458 16006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:23.130769 16006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:23.130893 16006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:23.138440 16006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:23.138568 16006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:23.138706 16006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:18:23.148165 16006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:46615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:23.149536 16006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:23.151393 16006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:23.164705 16011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:23.164748 16012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:23.164775 16014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:23.165673 16006 server_base.cc:1034] running on GCE node
I20260926 02:18:23.166304 16006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:23.167299 16006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:23.168529 16006 hybrid_clock.cc:648] HybridClock initialized: now 1790389103168469 us; error 51 us; skew 500 ppm
I20260926 02:18:23.168891 16006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:23.171159 16006 webserver.cc:460] Webserver started at http://127.28.152.131:36333/ using document root <none> and password file <none>
I20260926 02:18:23.171870 16006 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:23.171975 16006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:23.172297 16006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:23.175396 16006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "4be4bed0938544be923fc2087853d95b"
format_stamp: "Formatted at 2026-09-26 02:18:23 on dist-test-slave-hllj"
server_key: "bc2f8c6c85e988192e6ef9592f7f324e"
server_key_iv: "606190fc0409e83ae6b7ad03837baf89"
server_key_version: "encryptionkey@0"
I20260926 02:18:23.176232 16006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "4be4bed0938544be923fc2087853d95b"
format_stamp: "Formatted at 2026-09-26 02:18:23 on dist-test-slave-hllj"
server_key: "bc2f8c6c85e988192e6ef9592f7f324e"
server_key_iv: "606190fc0409e83ae6b7ad03837baf89"
server_key_version: "encryptionkey@0"
I20260926 02:18:23.181785 16006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:18:23.185628 16020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:23.187119 16006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:18:23.187274 16006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "4be4bed0938544be923fc2087853d95b"
format_stamp: "Formatted at 2026-09-26 02:18:23 on dist-test-slave-hllj"
server_key: "bc2f8c6c85e988192e6ef9592f7f324e"
server_key_iv: "606190fc0409e83ae6b7ad03837baf89"
server_key_version: "encryptionkey@0"
I20260926 02:18:23.187482 16006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:23.205868 16006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:23.206734 16006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:23.207036 16006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:23.208057 16006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:23.209600 16006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:23.209710 16006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:23.209816 16006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:23.209884 16006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:23.239778 16006 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:44265
I20260926 02:18:23.239822 16134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:44265 every 8 connection(s)
I20260926 02:18:23.241154 16006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:18:23.250605 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 16006
I20260926 02:18:23.250766 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:18:23.251197 29282 external_mini_cluster.cc:1428] Setting key 9605a646afc3a2330444d37305551864
I20260926 02:18:23.253275 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.243615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:32967 (local address 127.28.152.190:46615)
0926 02:18:23.243777 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:23.243781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:23.245161 (+ 1380us) server_negotiation.cc:407] Connection header received
0926 02:18:23.246916 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:23.246922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:23.247001 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:23.247090 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:23.248184 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.248919 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.250523 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.251007 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.251122 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.252447 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:23.252452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:23.252455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:23.252473 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:23.252505 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:23.252517 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:23.252587 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:23.252797 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 02:18:23.252917 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:18:23.254408 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.244096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:46615 (local address 127.28.152.131:32967)
0926 02:18:23.244982 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:23.245022 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:23.246627 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:23.247335 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:23.247353 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:23.247445 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:23.248016 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.248032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.249063 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.249067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.250385 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.250398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.251217 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.251224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.251477 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.252121 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:18:23.252124 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:23.252163 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:18:23.252248 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:23.252599 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:23.252611 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:23.252619 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:18:23.252774 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 02:18:23.253223 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":271,"threads_started":1}
I20260926 02:18:23.255640 16135 heartbeater.cc:347] Connected to a master server at 127.28.152.190:46615
I20260926 02:18:23.255894 16135 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:23.256619 16135 heartbeater.cc:511] Master 127.28.152.190:46615 requested a full tablet report, sending...
I20260926 02:18:23.257972 15685 ts_manager.cc:194] Registered new tserver with Master: 4be4bed0938544be923fc2087853d95b (127.28.152.131:44265)
I20260926 02:18:23.258812 15685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.131:32967
I20260926 02:18:23.266497 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:18:23.276937 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.270656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40082 (local address 127.28.152.190:46615)
0926 02:18:23.270915 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:23.270922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:23.271203 (+ 281us) server_negotiation.cc:407] Connection header received
0926 02:18:23.271379 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:23.271383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:23.271480 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:23.271570 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:23.272046 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.272796 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.274366 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.274800 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.274906 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.276080 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:23.276085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:23.276089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:23.276108 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:23.276147 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:23.276159 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:23.276291 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:23.276564 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 02:18:23.276631 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:18:23.281323 15685 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:18:23.288465 15685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:18:23.312731 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.301403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.28.152.131:44265)
0926 02:18:23.304702 (+ 3299us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:23.304712 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:23.304747 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:18:23.304847 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:23.304854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:23.305063 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:23.305263 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:23.306018 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.307639 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.309962 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.311372 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.311554 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.311735 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:23.312221 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 02:18:23.312407 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3114,"thread_start_us":2955,"threads_started":1}
I20260926 02:18:23.312817 16149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.301055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:44265 (local address 127.0.0.1:32778)
0926 02:18:23.302246 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:23.302304 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:23.302507 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:23.305269 (+ 2762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:23.305277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:23.305315 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:23.305796 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.305814 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.307784 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.307789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.309766 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.309779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.311833 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.311847 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.311997 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.312033 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:18:23.312142 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 02:18:23.312380 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":2,"thread_start_us":2145,"threads_started":1}
I20260926 02:18:23.313462 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.301441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:37091 (local address 127.0.0.1:42130)
0926 02:18:23.303511 (+ 2070us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:23.303530 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:23.303660 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:23.306384 (+ 2724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:23.306388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:23.306412 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:23.306812 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.306823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.308586 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.308591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.310857 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.310869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.312639 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.312647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.312885 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.312910 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:18:23.312968 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 02:18:23.313078 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1913,"mutex_wait_us":87,"thread_start_us":652,"threads_started":1}
I20260926 02:18:23.313786 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.303593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:46661 (local address 127.0.0.1:34096)
0926 02:18:23.304289 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:23.304305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:23.304409 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:23.305590 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:23.305600 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:23.305633 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:23.306056 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.306068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.308292 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.308300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.310558 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.310573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.312556 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.312564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.312804 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.312871 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 02:18:23.312935 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 02:18:23.313069 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":9,"thread_start_us":431,"threads_started":1}
I20260926 02:18:23.314411 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.304073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.28.152.130:46661)
0926 02:18:23.304893 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:23.304903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:23.304935 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:18:23.305047 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:23.305055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:23.305361 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:23.306702 (+ 1341us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:23.306944 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.308160 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.310815 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.312308 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.313595 (+ 1287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.313844 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:23.313919 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 02:18:23.314092 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":505,"threads_started":1}
I20260926 02:18:23.316274 16150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.301972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.28.152.129:37091)
0926 02:18:23.305899 (+ 3927us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:23.305908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:23.305936 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:18:23.306023 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:23.306027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:23.306211 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:23.306387 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:23.307024 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.308439 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.311074 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.312444 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.313914 (+ 1470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.315661 (+ 1747us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:23.315743 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 02:18:23.315925 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3768,"thread_start_us":3661,"threads_started":1}
I20260926 02:18:23.319087 16069 tablet_service.cc:1467] Processing CreateTablet for tablet c67d9a2a41904b5382245f4ffe62395e (DEFAULT_TABLE table=test-table [id=2980646ed2764aa68d5f6b02dc60f20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:23.319551 15937 tablet_service.cc:1467] Processing CreateTablet for tablet c67d9a2a41904b5382245f4ffe62395e (DEFAULT_TABLE table=test-table [id=2980646ed2764aa68d5f6b02dc60f20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:23.320061 15806 tablet_service.cc:1467] Processing CreateTablet for tablet c67d9a2a41904b5382245f4ffe62395e (DEFAULT_TABLE table=test-table [id=2980646ed2764aa68d5f6b02dc60f20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:23.320850 16069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67d9a2a41904b5382245f4ffe62395e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:23.321401 15937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67d9a2a41904b5382245f4ffe62395e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:23.321475 15806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67d9a2a41904b5382245f4ffe62395e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:23.333725 16153 tablet_bootstrap.cc:492] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: Bootstrap starting.
I20260926 02:18:23.336333 16154 tablet_bootstrap.cc:492] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf: Bootstrap starting.
I20260926 02:18:23.338095 16155 tablet_bootstrap.cc:492] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: Bootstrap starting.
I20260926 02:18:23.339156 16153 tablet_bootstrap.cc:654] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:23.340330 16154 tablet_bootstrap.cc:654] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:23.340530 16153 log.cc:824] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:23.341229 16155 tablet_bootstrap.cc:654] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:23.341563 16154 log.cc:824] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:23.342455 16155 log.cc:824] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:23.343876 16153 tablet_bootstrap.cc:492] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: No bootstrap required, opened a new log
I20260926 02:18:23.344189 16153 ts_tablet_manager.cc:1397] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260926 02:18:23.344708 16154 tablet_bootstrap.cc:492] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf: No bootstrap required, opened a new log
I20260926 02:18:23.344852 15871 heartbeater.cc:503] Master 127.28.152.190:46615 was elected leader, sending a full tablet report...
I20260926 02:18:23.345019 16154 ts_tablet_manager.cc:1397] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:18:23.345202 16155 tablet_bootstrap.cc:492] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: No bootstrap required, opened a new log
I20260926 02:18:23.345464 16155 ts_tablet_manager.cc:1397] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260926 02:18:23.354152 16154 raft_consensus.cc:348] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be4bed0938544be923fc2087853d95b" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 44265 } } peers { permanent_uuid: "3bab7c2a58f3440fa245b75adeca92bf" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 37091 } } peers { permanent_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46661 } }
I20260926 02:18:23.354583 16154 raft_consensus.cc:374] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:23.354679 16154 raft_consensus.cc:729] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bab7c2a58f3440fa245b75adeca92bf, State: Initialized, Role: FOLLOWER
I20260926 02:18:23.355307 16154 consensus_queue.cc:260] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be4bed0938544be923fc2087853d95b" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 44265 } } peers { permanent_uuid: "3bab7c2a58f3440fa245b75adeca92bf" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 37091 } } peers { permanent_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46661 } }
I20260926 02:18:23.356653 16153 raft_consensus.cc:348] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be4bed0938544be923fc2087853d95b" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 44265 } } peers { permanent_uuid: "3bab7c2a58f3440fa245b75adeca92bf" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 37091 } } peers { permanent_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46661 } }
I20260926 02:18:23.356653 16155 raft_consensus.cc:348] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be4bed0938544be923fc2087853d95b" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 44265 } } peers { permanent_uuid: "3bab7c2a58f3440fa245b75adeca92bf" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 37091 } } peers { permanent_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46661 } }
I20260926 02:18:23.357146 16153 raft_consensus.cc:374] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:23.357147 16155 raft_consensus.cc:374] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:23.357246 16153 raft_consensus.cc:729] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682f0e66099c4305b0cd7ec38d9d60c7, State: Initialized, Role: FOLLOWER
I20260926 02:18:23.357242 16155 raft_consensus.cc:729] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be4bed0938544be923fc2087853d95b, State: Initialized, Role: FOLLOWER
I20260926 02:18:23.357247 16154 ts_tablet_manager.cc:1428] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf: Time spent starting tablet: real 0.012s user 0.006s sys 0.008s
I20260926 02:18:23.357890 16155 consensus_queue.cc:260] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be4bed0938544be923fc2087853d95b" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 44265 } } peers { permanent_uuid: "3bab7c2a58f3440fa245b75adeca92bf" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 37091 } } peers { permanent_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46661 } }
I20260926 02:18:23.357895 16153 consensus_queue.cc:260] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be4bed0938544be923fc2087853d95b" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 44265 } } peers { permanent_uuid: "3bab7c2a58f3440fa245b75adeca92bf" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 37091 } } peers { permanent_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46661 } }
I20260926 02:18:23.361248 16153 ts_tablet_manager.cc:1428] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
I20260926 02:18:23.361585 16003 heartbeater.cc:503] Master 127.28.152.190:46615 was elected leader, sending a full tablet report...
I20260926 02:18:23.361670 16135 heartbeater.cc:503] Master 127.28.152.190:46615 was elected leader, sending a full tablet report...
I20260926 02:18:23.362123 16155 ts_tablet_manager.cc:1428] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
W20260926 02:18:23.492756 16136 tablet.cc:2369] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:18:23.532369 16004 tablet.cc:2369] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:18:23.571058 15872 tablet.cc:2369] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:18:23.572494 16160 raft_consensus.cc:482] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:18:23.572808 16160 raft_consensus.cc:504] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be4bed0938544be923fc2087853d95b" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 44265 } } peers { permanent_uuid: "3bab7c2a58f3440fa245b75adeca92bf" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 37091 } } peers { permanent_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46661 } }
I20260926 02:18:23.574601 16160 leader_election.cc:288] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4be4bed0938544be923fc2087853d95b (127.28.152.131:44265), 3bab7c2a58f3440fa245b75adeca92bf (127.28.152.129:37091)
I20260926 02:18:23.581760 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.575081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:48865 (local address 127.28.152.131:44265)
0926 02:18:23.575291 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:23.575298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:23.576014 (+ 716us) server_negotiation.cc:407] Connection header received
0926 02:18:23.576189 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:23.576195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:23.576267 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:23.576339 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:23.576964 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.577780 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.580056 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.580813 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.580964 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.581034 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:23.581403 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 02:18:23.581528 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:18:23.581768 16150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.575058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:33077 (local address 127.28.152.129:37091)
0926 02:18:23.575322 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:23.575330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:23.576103 (+ 773us) server_negotiation.cc:407] Connection header received
0926 02:18:23.576257 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:23.576260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:23.576341 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:23.576432 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:23.577041 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.577796 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.580008 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.580797 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.580952 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.581029 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:23.581360 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 02:18:23.581497 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 02:18:23.581892 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.575056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:44265 (local address 127.28.152.130:48865)
0926 02:18:23.575893 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:23.575909 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:23.576047 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:23.576369 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:23.576373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:23.576398 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:23.576812 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.576825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.577919 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.577971 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.579790 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.579804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.581029 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.581037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.581238 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.581262 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:18:23.581352 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 02:18:23.581497 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":130,"thread_start_us":498,"threads_started":1}
I20260926 02:18:23.582233 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.575140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:37091 (local address 127.28.152.130:33077)
0926 02:18:23.575985 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:23.576001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:23.576132 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:23.576463 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:23.576467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:23.576488 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:23.576886 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.576897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.577966 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.577971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.579792 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:23.579805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.581008 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.581015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:23.581192 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.581230 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:18:23.581305 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 02:18:23.581436 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":113,"thread_start_us":430,"threads_started":1}
I20260926 02:18:23.582996 16089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67d9a2a41904b5382245f4ffe62395e" candidate_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4be4bed0938544be923fc2087853d95b" is_pre_election: true
I20260926 02:18:23.583053 15826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67d9a2a41904b5382245f4ffe62395e" candidate_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bab7c2a58f3440fa245b75adeca92bf" is_pre_election: true
I20260926 02:18:23.583501 15826 raft_consensus.cc:2454] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 682f0e66099c4305b0cd7ec38d9d60c7 in term 0.
I20260926 02:18:23.583500 16089 raft_consensus.cc:2454] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 682f0e66099c4305b0cd7ec38d9d60c7 in term 0.
I20260926 02:18:23.584160 15892 leader_election.cc:302] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bab7c2a58f3440fa245b75adeca92bf, 682f0e66099c4305b0cd7ec38d9d60c7; no voters:
I20260926 02:18:23.584548 16160 raft_consensus.cc:2781] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:18:23.584672 16160 raft_consensus.cc:482] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:18:23.584771 16160 raft_consensus.cc:3037] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:23.587025 16160 raft_consensus.cc:504] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be4bed0938544be923fc2087853d95b" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 44265 } } peers { permanent_uuid: "3bab7c2a58f3440fa245b75adeca92bf" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 37091 } } peers { permanent_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46661 } }
I20260926 02:18:23.587680 16160 leader_election.cc:288] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [CANDIDATE]: Term 1 election: Requested vote from peers 4be4bed0938544be923fc2087853d95b (127.28.152.131:44265), 3bab7c2a58f3440fa245b75adeca92bf (127.28.152.129:37091)
I20260926 02:18:23.588253 16089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67d9a2a41904b5382245f4ffe62395e" candidate_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4be4bed0938544be923fc2087853d95b"
I20260926 02:18:23.588303 15826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67d9a2a41904b5382245f4ffe62395e" candidate_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bab7c2a58f3440fa245b75adeca92bf"
I20260926 02:18:23.588459 16089 raft_consensus.cc:3037] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:23.588495 15826 raft_consensus.cc:3037] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:23.590842 16089 raft_consensus.cc:2454] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 682f0e66099c4305b0cd7ec38d9d60c7 in term 1.
I20260926 02:18:23.590848 15826 raft_consensus.cc:2454] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 682f0e66099c4305b0cd7ec38d9d60c7 in term 1.
I20260926 02:18:23.591367 15892 leader_election.cc:302] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bab7c2a58f3440fa245b75adeca92bf, 682f0e66099c4305b0cd7ec38d9d60c7; no voters:
I20260926 02:18:23.591714 16160 raft_consensus.cc:2781] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:23.592187 16160 raft_consensus.cc:686] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 1 LEADER]: Becoming Leader. State: Replica: 682f0e66099c4305b0cd7ec38d9d60c7, State: Running, Role: LEADER
I20260926 02:18:23.592684 16160 consensus_queue.cc:237] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be4bed0938544be923fc2087853d95b" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 44265 } } peers { permanent_uuid: "3bab7c2a58f3440fa245b75adeca92bf" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 37091 } } peers { permanent_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46661 } }
I20260926 02:18:23.597795 15684 catalog_manager.cc:5495] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 682f0e66099c4305b0cd7ec38d9d60c7 (127.28.152.130). New cstate: current_term: 1 leader_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4be4bed0938544be923fc2087853d95b" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 44265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bab7c2a58f3440fa245b75adeca92bf" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 37091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "682f0e66099c4305b0cd7ec38d9d60c7" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 46661 } health_report { overall_health: HEALTHY } } }
I20260926 02:18:23.618244 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.612559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.28.152.130:46661)
0926 02:18:23.612802 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:23.612806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:23.612826 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:18:23.612951 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:23.612956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:23.613049 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:23.613229 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:18:23.613645 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.614372 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.615849 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.616285 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.616391 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.616957 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:18:23.617605 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:18:23.617767 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:23.617909 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 02:18:23.617953 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 02:18:23.626969 15826 raft_consensus.cc:1261] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf [term 1 FOLLOWER]: Refusing update from remote peer 682f0e66099c4305b0cd7ec38d9d60c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:23.626971 16089 raft_consensus.cc:1261] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b [term 1 FOLLOWER]: Refusing update from remote peer 682f0e66099c4305b0cd7ec38d9d60c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:23.627982 16164 consensus_queue.cc:1035] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4be4bed0938544be923fc2087853d95b" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:23.628319 16160 consensus_queue.cc:1035] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bab7c2a58f3440fa245b75adeca92bf" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:23.643692 16168 mvcc.cc:205] Tried to move back new op lower bound from 7333433768445997056 to 7333433768319868928. Current Snapshot: MvccSnapshot[applied={T|T < 7333433768445997056}]
I20260926 02:18:23.655229 15684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:23.655494 15684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:23.659269 15684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a8c2af3e82747cba079e7773316ea88: Sending DeleteTablet for 3 replicas of tablet c67d9a2a41904b5382245f4ffe62395e
I20260926 02:18:23.660557 15806 tablet_service.cc:1514] Processing DeleteTablet for tablet c67d9a2a41904b5382245f4ffe62395e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:23 UTC) from {username='slave'} at 127.0.0.1:42130
I20260926 02:18:23.660781 16069 tablet_service.cc:1514] Processing DeleteTablet for tablet c67d9a2a41904b5382245f4ffe62395e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:23 UTC) from {username='slave'} at 127.0.0.1:32778
I20260926 02:18:23.661361 15937 tablet_service.cc:1514] Processing DeleteTablet for tablet c67d9a2a41904b5382245f4ffe62395e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:23 UTC) from {username='slave'} at 127.0.0.1:34096
I20260926 02:18:23.661899 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 15744
I20260926 02:18:23.672610 16180 tablet_replica.cc:330] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: stopping tablet replica
I20260926 02:18:23.674027 16179 tablet_replica.cc:330] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: stopping tablet replica
I20260926 02:18:23.674672 16179 raft_consensus.cc:2229] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:23.674919 16180 raft_consensus.cc:2229] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:18:23.675380 16179 raft_consensus.cc:2258] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:23.675643 16180 raft_consensus.cc:2258] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:23.676280 16178 tablet_replica.cc:330] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf: stopping tablet replica
I20260926 02:18:23.676687 16150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.663110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.28.152.129:37091)
0926 02:18:23.663300 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:23.663305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:23.665578 (+ 2273us) server_negotiation.cc:407] Connection header received
0926 02:18:23.665864 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:23.665871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:23.666027 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:23.666109 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:23.666769 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.668076 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.670290 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.671012 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.671114 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.673632 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:23.673645 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:23.673664 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:23.674495 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:23.674561 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:23.674589 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:23.674677 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:23.676099 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0926 02:18:23.676224 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:18:23.678103 15866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:23.678529 16180 ts_tablet_manager.cc:1905] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:23.680172 16179 ts_tablet_manager.cc:1905] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:23.687695 16179 ts_tablet_manager.cc:1918] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:23.687700 16180 ts_tablet_manager.cc:1918] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:23.687906 16180 log.cc:1196] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/c67d9a2a41904b5382245f4ffe62395e
I20260926 02:18:23.688086 16179 log.cc:1196] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/c67d9a2a41904b5382245f4ffe62395e
I20260926 02:18:23.688529 16180 ts_tablet_manager.cc:1939] T c67d9a2a41904b5382245f4ffe62395e P 682f0e66099c4305b0cd7ec38d9d60c7: Deleting consensus metadata
I20260926 02:18:23.688717 16179 ts_tablet_manager.cc:1939] T c67d9a2a41904b5382245f4ffe62395e P 4be4bed0938544be923fc2087853d95b: Deleting consensus metadata
I20260926 02:18:23.690568 15669 catalog_manager.cc:4841] TS 4be4bed0938544be923fc2087853d95b (127.28.152.131:44265): tablet c67d9a2a41904b5382245f4ffe62395e (table test-table [id=2980646ed2764aa68d5f6b02dc60f20d]) successfully deleted
I20260926 02:18:23.690908 15669 catalog_manager.cc:4841] TS 682f0e66099c4305b0cd7ec38d9d60c7 (127.28.152.130:46661): tablet c67d9a2a41904b5382245f4ffe62395e (table test-table [id=2980646ed2764aa68d5f6b02dc60f20d]) successfully deleted
I20260926 02:18:23.679036 16178 raft_consensus.cc:2229] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:23.801817 16178 raft_consensus.cc:2258] T c67d9a2a41904b5382245f4ffe62395e P 3bab7c2a58f3440fa245b75adeca92bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:23.802022 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 15744
W20260926 02:18:23.820233 15671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.152.129:37091 (error 108)
I20260926 02:18:23.820888 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 15875
I20260926 02:18:23.823167 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.821640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:37091 (local address 127.0.0.1:42152)
0926 02:18:23.822885 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:23.823004 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:37091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1119,"thread_start_us":1001,"threads_started":1}
W20260926 02:18:23.823463 15671 catalog_manager.cc:4584] TS 3bab7c2a58f3440fa245b75adeca92bf (127.28.152.129:37091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c67d9a2a41904b5382245f4ffe62395e: Network error: Client connection negotiation failed: client connection to 127.28.152.129:37091: connect: Connection refused (error 111)
I20260926 02:18:23.829392 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.821676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.28.152.130:46661)
0926 02:18:23.822190 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:23.822194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:23.822210 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:18:23.822263 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:23.822267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:23.822349 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:23.822436 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:23.822917 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.824134 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.825624 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.826308 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.826450 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.827613 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:23.827626 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:23.827645 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:23.828506 (+ 861us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:23.828572 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:23.828598 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:23.828710 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:23.828828 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 02:18:23.828950 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398}
I20260926 02:18:23.830253 15997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:23.877077 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.876561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:37091 (local address 127.0.0.1:42158)
0926 02:18:23.876814 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:23.876918 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:37091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 02:18:23.878464 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.877973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:37091 (local address 127.0.0.1:42174)
0926 02:18:23.878150 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:23.878299 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:37091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 02:18:23.939435 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.938889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:37091 (local address 127.0.0.1:42186)
0926 02:18:23.939154 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:23.939258 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:37091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260926 02:18:23.940466 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.940139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:37091 (local address 127.0.0.1:42200)
0926 02:18:23.940254 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:23.940332 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:37091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 02:18:23.944675 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 15875
I20260926 02:18:23.960435 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 16006
I20260926 02:18:23.967705 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:23.961215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32786 (local address 127.28.152.131:44265)
0926 02:18:23.961397 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:23.961404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:23.961500 (+ 96us) server_negotiation.cc:407] Connection header received
0926 02:18:23.961685 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:23.961691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:23.961804 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:23.961892 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:23.962298 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.963032 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.964496 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:23.964904 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:23.965013 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:23.966352 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:23.966362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:23.966377 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:23.966951 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:23.967001 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:23.967022 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:23.967124 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:23.967260 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 02:18:23.967380 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 02:18:23.968401 16130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:24.021353 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:24.020765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:37091 (local address 127.0.0.1:42202)
0926 02:18:24.021057 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:24.021200 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:37091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260926 02:18:24.022446 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:24.022004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:37091 (local address 127.0.0.1:42204)
0926 02:18:24.022217 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:24.022288 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:37091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 02:18:24.055660 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 16006
I20260926 02:18:24.069909 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 15654
I20260926 02:18:24.071015 15715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:24.177006 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 15654
2026-09-26T02:18:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2941 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:24 dist-test-slave-hllj krb5kdc[16190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:24 dist-test-slave-hllj krb5kdc[16190](info): set up 2 sockets
Sep 26 02:18:24 dist-test-slave-hllj krb5kdc[16190](info): commencing operation
krb5kdc: starting...
W20260926 02:18:25.206650 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:25 dist-test-slave-hllj krb5kdc[16190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:18:25.456687 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:41393
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:41393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:25.790532 16203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:25.790855 16203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:25.790993 16203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:25.798377 16203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:25.798514 16203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:25.798593 16203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:25.798667 16203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:25.798731 16203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:25.808641 16203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:41393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:41393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:25.810051 16203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:25.811432 16203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:25.823505 16208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:25.823474 16211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:25.823535 16209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:25.824357 16203 server_base.cc:1034] running on GCE node
I20260926 02:18:25.825258 16203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:25.826753 16203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:25.827975 16203 hybrid_clock.cc:648] HybridClock initialized: now 1790389105827933 us; error 38 us; skew 500 ppm
I20260926 02:18:25.828382 16203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:25 dist-test-slave-hllj krb5kdc[16190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389105, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:25.838917 16203 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:18:25.840852 16203 webserver.cc:460] Webserver started at http://127.28.152.190:39577/ using document root <none> and password file <none>
I20260926 02:18:25.841651 16203 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:25.841780 16203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:25.842080 16203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:25.845268 16203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "86e020ecbfbc4a0b809c8f47e9009a54"
format_stamp: "Formatted at 2026-09-26 02:18:25 on dist-test-slave-hllj"
server_key: "9f06166dc131e94c6480b15f9555e3f2"
server_key_iv: "e6a777720c9c5bd7018e62464ca1425a"
server_key_version: "encryptionkey@0"
I20260926 02:18:25.846066 16203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "86e020ecbfbc4a0b809c8f47e9009a54"
format_stamp: "Formatted at 2026-09-26 02:18:25 on dist-test-slave-hllj"
server_key: "9f06166dc131e94c6480b15f9555e3f2"
server_key_iv: "e6a777720c9c5bd7018e62464ca1425a"
server_key_version: "encryptionkey@0"
I20260926 02:18:25.851629 16203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:18:25.855286 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:25.856694 16203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:18:25.856859 16203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "86e020ecbfbc4a0b809c8f47e9009a54"
format_stamp: "Formatted at 2026-09-26 02:18:25 on dist-test-slave-hllj"
server_key: "9f06166dc131e94c6480b15f9555e3f2"
server_key_iv: "e6a777720c9c5bd7018e62464ca1425a"
server_key_version: "encryptionkey@0"
I20260926 02:18:25.857045 16203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:25.885480 16203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:25.890018 16203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:25.890347 16203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:25.908300 16203 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:41393
I20260926 02:18:25.908319 16269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:41393 every 8 connection(s)
I20260926 02:18:25.909873 16203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:25.910960 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 16203
I20260926 02:18:25.911154 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:25.911688 29282 external_mini_cluster.cc:1428] Setting key b52c3c47eb1bc3664eaa9b75bf7fc9d8
I20260926 02:18:25.915432 16270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:18:25 dist-test-slave-hllj krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:25.928520 16270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54: Bootstrap starting.
I20260926 02:18:25.932029 16270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:25.933202 16270 log.cc:824] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:25.936129 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:25.913575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.28.152.190:41393)
0926 02:18:25.914374 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:25.914390 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:25.914438 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:18:25.916154 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:25.916178 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:25.916591 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:25.916950 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:25.918789 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:25.920181 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:25.922085 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:25.922781 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:25.922931 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:25.927807 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:25.927815 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:25.927833 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:25.927886 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:25.931190 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:25.932064 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:25.932072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:25.932081 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:25.932324 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:25.932749 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:25.932753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:25.932755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:25.933474 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:25.933796 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:25.934282 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 02:18:25.934721 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":338,"threads_started":1}
I20260926 02:18:25.936930 16270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54: No bootstrap required, opened a new log
I20260926 02:18:25.945739 16270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e020ecbfbc4a0b809c8f47e9009a54" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41393 } }
I20260926 02:18:25.946142 16270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:25.946247 16270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e020ecbfbc4a0b809c8f47e9009a54, State: Initialized, Role: FOLLOWER
I20260926 02:18:25.946827 16270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e020ecbfbc4a0b809c8f47e9009a54" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41393 } }
I20260926 02:18:25.947041 16270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:25.947165 16270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:25.947321 16270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:25.949684 16270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e020ecbfbc4a0b809c8f47e9009a54" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41393 } }
I20260926 02:18:25.950224 16270 leader_election.cc:302] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86e020ecbfbc4a0b809c8f47e9009a54; no voters:
I20260926 02:18:25.950821 16270 leader_election.cc:288] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:25.950973 16275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:25.951606 16275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [term 1 LEADER]: Becoming Leader. State: Replica: 86e020ecbfbc4a0b809c8f47e9009a54, State: Running, Role: LEADER
I20260926 02:18:25.952162 16275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e020ecbfbc4a0b809c8f47e9009a54" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41393 } }
I20260926 02:18:25.952907 16270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:25.955071 16277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86e020ecbfbc4a0b809c8f47e9009a54. Latest consensus state: current_term: 1 leader_uuid: "86e020ecbfbc4a0b809c8f47e9009a54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e020ecbfbc4a0b809c8f47e9009a54" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41393 } } }
I20260926 02:18:25.955469 16277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:25.955489 16276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86e020ecbfbc4a0b809c8f47e9009a54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e020ecbfbc4a0b809c8f47e9009a54" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 41393 } } }
I20260926 02:18:25.956254 16276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:25.958451 16284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:25.963166 16284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:25.972129 16284 catalog_manager.cc:1347] Generated new cluster ID: 2dbead50fc704f4593520a611fd8d967
I20260926 02:18:25.972262 16284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:25.998309 16284 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:26.000319 16284 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:26.010524 16284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54: Generated new TSK 0
I20260926 02:18:26.011672 16284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.152.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
Entry for principal kudu/127.28.152.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab.
I20260926 02:18:26.080550 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41393
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:26.417047 16296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:26.417403 16296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:26.417541 16296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:26.425335 16296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:26.425464 16296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:26.425617 16296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:18:26.436228 16296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.129.keytab
--principal=kudu/127.28.152.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:26.437826 16296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:26.439208 16296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:26.452430 16301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:26.452497 16304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:26.452526 16302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:26.453421 16296 server_base.cc:1034] running on GCE node
I20260926 02:18:26.454131 16296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:26.455152 16296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:26.456364 16296 hybrid_clock.cc:648] HybridClock initialized: now 1790389106456310 us; error 37 us; skew 500 ppm
I20260926 02:18:26.456720 16296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:26 dist-test-slave-hllj krb5kdc[16190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389106, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:26.459969 16296 init.cc:377] Logged in from keytab as kudu/127.28.152.129@KRBTEST.COM (short username kudu)
I20260926 02:18:26.461733 16296 webserver.cc:460] Webserver started at http://127.28.152.129:42211/ using document root <none> and password file <none>
I20260926 02:18:26.462494 16296 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:26.462606 16296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:26.462946 16296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:26.466120 16296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "ef2b521a277f42b2bc1d4848fc9d7923"
format_stamp: "Formatted at 2026-09-26 02:18:26 on dist-test-slave-hllj"
server_key: "94c4b12198c50c2aa725396a96890633"
server_key_iv: "c15f7f24e238c70708d4744c652cd209"
server_key_version: "encryptionkey@0"
I20260926 02:18:26.466928 16296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "ef2b521a277f42b2bc1d4848fc9d7923"
format_stamp: "Formatted at 2026-09-26 02:18:26 on dist-test-slave-hllj"
server_key: "94c4b12198c50c2aa725396a96890633"
server_key_iv: "c15f7f24e238c70708d4744c652cd209"
server_key_version: "encryptionkey@0"
I20260926 02:18:26.472476 16296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:18:26.476289 16311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:26.477785 16296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:18:26.477960 16296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "ef2b521a277f42b2bc1d4848fc9d7923"
format_stamp: "Formatted at 2026-09-26 02:18:26 on dist-test-slave-hllj"
server_key: "94c4b12198c50c2aa725396a96890633"
server_key_iv: "c15f7f24e238c70708d4744c652cd209"
server_key_version: "encryptionkey@0"
I20260926 02:18:26.478159 16296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:26.508975 16296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:26.513907 16296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:26.514185 16296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:26.515188 16296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:26.516664 16296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:26.516739 16296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:26.516852 16296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:26.516917 16296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:26.547837 16296 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:45085
I20260926 02:18:26.547899 16424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:45085 every 8 connection(s)
I20260926 02:18:26.549201 16296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:18:26.554983 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 16296
I20260926 02:18:26.555152 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:18:26.555634 29282 external_mini_cluster.cc:1428] Setting key beee9b0bb2ef26008d0f1340bca32c19
Sep 26 02:18:26 dist-test-slave-hllj krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389106, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.129@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:26.571907 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:26.551679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:45049 (local address 127.28.152.190:41393)
0926 02:18:26.552115 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:26.552119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:26.553085 (+ 966us) server_negotiation.cc:407] Connection header received
0926 02:18:26.555033 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:26.555039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:26.555128 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:26.555248 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:26.557025 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:26.557958 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:26.560226 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:26.560698 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:26.560798 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:26.565219 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:26.565228 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:26.565231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:26.565269 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:26.568433 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:26.569311 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:26.569318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:26.569322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:26.569480 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:26.570020 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:26.570024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:26.570027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:26.570390 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:26.570540 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:26.571332 (+ 792us) server_negotiation.cc:299] Negotiation successful
0926 02:18:26.571569 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":232,"threads_started":1}
I20260926 02:18:26.572806 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:26.552162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:41393 (local address 127.28.152.129:45049)
0926 02:18:26.552924 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:26.552963 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:26.554745 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:26.555582 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:26.555592 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:26.556195 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:26.556834 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:26.556854 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:26.558147 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:26.558152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:26.560092 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:26.560106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:26.560923 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:26.560930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:26.561245 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:26.562055 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:18:26.562082 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:26.564839 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:26.568636 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:26.568645 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:26.568662 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:26.569177 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:26.569637 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:26.569641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:26.569643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:26.569895 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:26.570514 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:26.570521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:26.570921 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 02:18:26.571163 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 02:18:26.571602 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":289,"threads_started":1}
I20260926 02:18:26.574616 16425 heartbeater.cc:347] Connected to a master server at 127.28.152.190:41393
I20260926 02:18:26.574981 16425 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:26.575883 16425 heartbeater.cc:511] Master 127.28.152.190:41393 requested a full tablet report, sending...
I20260926 02:18:26.578354 16235 ts_manager.cc:194] Registered new tserver with Master: ef2b521a277f42b2bc1d4848fc9d7923 (127.28.152.129:45085)
I20260926 02:18:26.579916 16235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.129@KRBTEST.COM'} at 127.28.152.129:45049
WARNING: no policy specified for kudu/127.28.152.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
Entry for principal kudu/127.28.152.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab.
I20260926 02:18:26.612007 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41393
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:26.946233 16431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:26.946557 16431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:26.946698 16431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:26.954263 16431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:26.954407 16431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:26.954558 16431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:18:26.965049 16431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.130.keytab
--principal=kudu/127.28.152.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:26.966451 16431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:26.967854 16431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:26.981771 16439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:26.981971 16437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:26.982342 16436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:26.986836 16431 server_base.cc:1034] running on GCE node
I20260926 02:18:26.987527 16431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:26.988565 16431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:26.989792 16431 hybrid_clock.cc:648] HybridClock initialized: now 1790389106989727 us; error 48 us; skew 500 ppm
I20260926 02:18:26.990139 16431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:26 dist-test-slave-hllj krb5kdc[16190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389106, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:26.993432 16431 init.cc:377] Logged in from keytab as kudu/127.28.152.130@KRBTEST.COM (short username kudu)
I20260926 02:18:26.995018 16431 webserver.cc:460] Webserver started at http://127.28.152.130:43295/ using document root <none> and password file <none>
I20260926 02:18:26.995787 16431 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:26.995903 16431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:26.996197 16431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:26.999168 16431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "776daa1d4878453c8af3d8902067ea7f"
format_stamp: "Formatted at 2026-09-26 02:18:26 on dist-test-slave-hllj"
server_key: "5599c5eb681ac3fd18b0a9a4a012d56f"
server_key_iv: "742557eac19c723a51d3def6d6ff6dc7"
server_key_version: "encryptionkey@0"
I20260926 02:18:26.999939 16431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "776daa1d4878453c8af3d8902067ea7f"
format_stamp: "Formatted at 2026-09-26 02:18:26 on dist-test-slave-hllj"
server_key: "5599c5eb681ac3fd18b0a9a4a012d56f"
server_key_iv: "742557eac19c723a51d3def6d6ff6dc7"
server_key_version: "encryptionkey@0"
I20260926 02:18:27.005503 16431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 02:18:27.009368 16446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:27.011087 16431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:27.011264 16431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "776daa1d4878453c8af3d8902067ea7f"
format_stamp: "Formatted at 2026-09-26 02:18:26 on dist-test-slave-hllj"
server_key: "5599c5eb681ac3fd18b0a9a4a012d56f"
server_key_iv: "742557eac19c723a51d3def6d6ff6dc7"
server_key_version: "encryptionkey@0"
I20260926 02:18:27.011489 16431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:27.047325 16431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:27.051820 16431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:27.052104 16431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:27.053115 16431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:27.054595 16431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:27.054685 16431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:27.054804 16431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:27.054865 16431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:27.086228 16431 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:34147
I20260926 02:18:27.086306 16559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:34147 every 8 connection(s)
I20260926 02:18:27.087574 16431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:18:27.096944 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 16431
I20260926 02:18:27.097113 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:18:27.097580 29282 external_mini_cluster.cc:1428] Setting key 7fb3efc14230e9d7329a838e8a38ff45
Sep 26 02:18:27 dist-test-slave-hllj krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389106, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.130@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:27.107820 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.090190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:38805 (local address 127.28.152.190:41393)
0926 02:18:27.090644 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:27.090649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:27.091649 (+ 1000us) server_negotiation.cc:407] Connection header received
0926 02:18:27.093606 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:27.093612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:27.093696 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:27.093792 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:27.095394 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.096128 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.097926 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.098400 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.098516 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.101981 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:27.101987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:27.101990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:27.102023 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:27.104592 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:27.105404 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:27.105408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:27.105410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:27.105544 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:27.105981 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:27.105985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:27.105987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:27.106290 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:27.106428 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:27.107177 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 02:18:27.107448 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":222,"threads_started":1}
I20260926 02:18:27.108848 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.090786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:41393 (local address 127.28.152.130:38805)
0926 02:18:27.091478 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:27.091521 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:27.093309 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:27.094050 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:27.094059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:27.094594 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:27.095205 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.095227 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.096286 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.096290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.097768 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.097783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.098624 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.098632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.098887 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.099541 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:18:27.099565 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:27.101746 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:27.104741 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:27.104759 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:27.104776 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:27.105279 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:27.105652 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:27.105657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:27.105661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:27.105864 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:27.106415 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:27.106423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:27.106799 (+ 376us) client_negotiation.cc:769] Sending connection context
0926 02:18:27.107052 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 02:18:27.107620 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":274,"threads_started":1}
I20260926 02:18:27.110411 16560 heartbeater.cc:347] Connected to a master server at 127.28.152.190:41393
I20260926 02:18:27.110785 16560 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:27.111671 16560 heartbeater.cc:511] Master 127.28.152.190:41393 requested a full tablet report, sending...
I20260926 02:18:27.113389 16235 ts_manager.cc:194] Registered new tserver with Master: 776daa1d4878453c8af3d8902067ea7f (127.28.152.130:34147)
I20260926 02:18:27.114233 16235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.130@KRBTEST.COM'} at 127.28.152.130:38805
WARNING: no policy specified for kudu/127.28.152.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
Entry for principal kudu/127.28.152.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab.
I20260926 02:18:27.152281 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41393
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:27.482014 16566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:27.482331 16566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:27.482460 16566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:27.489887 16566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:27.490008 16566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:27.490141 16566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:18:27.500114 16566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.131.keytab
--principal=kudu/127.28.152.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:41393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:27.501430 16566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:27.502761 16566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:27.515904 16572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:27.516177 16571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:27.516546 16574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:27.516986 16566 server_base.cc:1034] running on GCE node
I20260926 02:18:27.517614 16566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:27.518586 16566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:27.519821 16566 hybrid_clock.cc:648] HybridClock initialized: now 1790389107519747 us; error 54 us; skew 500 ppm
I20260926 02:18:27.520187 16566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:27 dist-test-slave-hllj krb5kdc[16190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:27.523710 16566 init.cc:377] Logged in from keytab as kudu/127.28.152.131@KRBTEST.COM (short username kudu)
I20260926 02:18:27.525373 16566 webserver.cc:460] Webserver started at http://127.28.152.131:33091/ using document root <none> and password file <none>
I20260926 02:18:27.526094 16566 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:27.526288 16566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:27.526600 16566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:27.529723 16566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "4952199e44014a8ab139abc350f52d1e"
format_stamp: "Formatted at 2026-09-26 02:18:27 on dist-test-slave-hllj"
server_key: "23e9e94ac497cdce47698ff8ec9c1256"
server_key_iv: "11ded51905bccef3d38f172ebdf44dfc"
server_key_version: "encryptionkey@0"
I20260926 02:18:27.530526 16566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "4952199e44014a8ab139abc350f52d1e"
format_stamp: "Formatted at 2026-09-26 02:18:27 on dist-test-slave-hllj"
server_key: "23e9e94ac497cdce47698ff8ec9c1256"
server_key_iv: "11ded51905bccef3d38f172ebdf44dfc"
server_key_version: "encryptionkey@0"
I20260926 02:18:27.536171 16566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:18:27.539947 16581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:27.541610 16566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:18:27.541788 16566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "4952199e44014a8ab139abc350f52d1e"
format_stamp: "Formatted at 2026-09-26 02:18:27 on dist-test-slave-hllj"
server_key: "23e9e94ac497cdce47698ff8ec9c1256"
server_key_iv: "11ded51905bccef3d38f172ebdf44dfc"
server_key_version: "encryptionkey@0"
I20260926 02:18:27.541998 16566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:27.566355 16566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:27.570914 16566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:27.571275 16566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:27.572273 16566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:27.573681 16566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:27.573767 16566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:27.573890 16566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:27.573943 16566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:27.583858 16425 heartbeater.cc:503] Master 127.28.152.190:41393 was elected leader, sending a full tablet report...
I20260926 02:18:27.604038 16566 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:46731
I20260926 02:18:27.604074 16694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:46731 every 8 connection(s)
I20260926 02:18:27.605384 16566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:18:27.607136 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 16566
I20260926 02:18:27.607319 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:18:27.607815 29282 external_mini_cluster.cc:1428] Setting key 09c3c360eebde7e46d43a5d2c6b6387c
Sep 26 02:18:27 dist-test-slave-hllj krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.131@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:27.626035 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.608621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:40683 (local address 127.28.152.190:41393)
0926 02:18:27.609064 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:27.609070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:27.610118 (+ 1048us) server_negotiation.cc:407] Connection header received
0926 02:18:27.612055 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:27.612060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:27.612141 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:27.612231 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:27.613820 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.614642 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.616258 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.616671 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.616786 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.620184 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:27.620189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:27.620192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:27.620225 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:27.623081 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:27.623773 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:27.623779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:27.623782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:27.623908 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:27.624318 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:27.624322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:27.624324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:27.624588 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:27.624723 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:27.625445 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 02:18:27.625665 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":209,"threads_started":1}
I20260926 02:18:27.627019 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.608825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:41393 (local address 127.28.152.131:40683)
0926 02:18:27.609976 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:27.610014 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:27.611805 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:27.612493 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:27.612503 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:27.613001 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:27.613649 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.613668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.614788 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.614792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.616126 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.616138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.616879 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.616887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.617112 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.617767 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:18:27.617789 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:27.619964 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:27.623208 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:27.623215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:27.623229 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:27.623650 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:27.624015 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:18:27.624018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:18:27.624021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:18:27.624202 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:18:27.624705 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:27.624713 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:27.625055 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 02:18:27.625342 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 02:18:27.625901 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":350,"threads_started":1}
I20260926 02:18:27.628329 16695 heartbeater.cc:347] Connected to a master server at 127.28.152.190:41393
I20260926 02:18:27.628647 16695 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:27.629432 16695 heartbeater.cc:511] Master 127.28.152.190:41393 requested a full tablet report, sending...
I20260926 02:18:27.631022 16235 ts_manager.cc:194] Registered new tserver with Master: 4952199e44014a8ab139abc350f52d1e (127.28.152.131:46731)
I20260926 02:18:27.631834 16235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.152.131@KRBTEST.COM'} at 127.28.152.131:40683
I20260926 02:18:27.635741 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:18:27.653363 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.639980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.28.152.190:41393)
0926 02:18:27.640182 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:27.640190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:27.640601 (+ 411us) server_negotiation.cc:407] Connection header received
0926 02:18:27.640758 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:27.640761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:27.640829 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:27.640908 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:27.642461 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.643311 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.644664 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.645073 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.645221 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.647618 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:27.647623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:27.647626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:27.647659 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:27.650319 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:27.651054 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:27.651059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:27.651061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:27.651269 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:27.651745 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:27.651750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:27.651752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:27.652103 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:27.652311 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:27.652768 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 02:18:27.652963 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:18:27.657466 16235 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:18:27.664701 16235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:18:27.701417 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.692678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.28.152.131:46731)
0926 02:18:27.693737 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:27.693744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:27.694478 (+ 734us) server_negotiation.cc:407] Connection header received
0926 02:18:27.694619 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:27.694626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:27.694808 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:27.694965 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:27.695592 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.696934 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.698831 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.700229 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.700418 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.700610 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:27.700894 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 02:18:27.701064 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":767,"threads_started":1}
I20260926 02:18:27.701665 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.678429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.28.152.130:34147)
0926 02:18:27.691203 (+ 12774us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:27.691212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:27.692373 (+ 1161us) server_negotiation.cc:407] Connection header received
0926 02:18:27.692478 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:27.692486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:27.692770 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:27.692963 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:27.694259 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.695822 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.699260 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.700319 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.700480 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.700647 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:27.701204 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 02:18:27.701377 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12629,"thread_start_us":12504,"threads_started":1}
I20260926 02:18:27.701949 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.678353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.28.152.129:45085)
0926 02:18:27.690748 (+ 12395us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:27.690761 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:27.692371 (+ 1610us) server_negotiation.cc:407] Connection header received
0926 02:18:27.692562 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:27.692569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:27.692786 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:27.692963 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:27.694160 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.695826 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.699040 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.700449 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.700637 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.700790 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:27.701476 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 02:18:27.701631 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12155,"thread_start_us":12862,"threads_started":1}
I20260926 02:18:27.702054 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.678190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45085 (local address 127.0.0.1:42030)
0926 02:18:27.691813 (+ 13623us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:27.691893 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:27.692144 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:27.693261 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:27.693270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:27.693314 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:27.693941 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.693956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.696000 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.696006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.698887 (+ 2881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.698896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.700940 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.700946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.701102 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.701167 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 02:18:27.701239 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 02:18:27.701438 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13505,"mutex_wait_us":178,"spinlock_wait_cycles":21632,"thread_start_us":13359,"threads_started":1}
I20260926 02:18:27.702521 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.677754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:34147 (local address 127.0.0.1:60506)
0926 02:18:27.691813 (+ 14059us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:27.691893 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:27.692144 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:27.693261 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:27.693270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:27.693312 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:27.694014 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.694024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.696000 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.696006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.698878 (+ 2872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.698894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.700806 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.700812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.700991 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.701023 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:18:27.701851 (+ 828us) client_negotiation.cc:240] Negotiation successful
0926 02:18:27.701996 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13742,"mutex_wait_us":298,"spinlock_wait_cycles":11392,"thread_start_us":13602,"threads_started":1}
I20260926 02:18:27.702580 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.693779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:46731 (local address 127.0.0.1:57276)
0926 02:18:27.694363 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:27.694378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:27.694492 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:27.694978 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:27.694982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:27.695005 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:27.695393 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.695405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.697080 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.697084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.698674 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.698687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.700420 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.700426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.700610 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.700658 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 02:18:27.701997 (+ 1339us) client_negotiation.cc:240] Negotiation successful
0926 02:18:27.702193 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":22,"thread_start_us":321,"threads_started":1}
I20260926 02:18:27.706414 16495 tablet_service.cc:1467] Processing CreateTablet for tablet d9597e28988c41b88feef31ee80f0407 (DEFAULT_TABLE table=test-table [id=d5e5fbf8eeb64cfb91d0f4e74a30adee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:27.706421 16360 tablet_service.cc:1467] Processing CreateTablet for tablet d9597e28988c41b88feef31ee80f0407 (DEFAULT_TABLE table=test-table [id=d5e5fbf8eeb64cfb91d0f4e74a30adee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:27.707978 16495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9597e28988c41b88feef31ee80f0407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:27.707975 16360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9597e28988c41b88feef31ee80f0407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:27.709749 16630 tablet_service.cc:1467] Processing CreateTablet for tablet d9597e28988c41b88feef31ee80f0407 (DEFAULT_TABLE table=test-table [id=d5e5fbf8eeb64cfb91d0f4e74a30adee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:27.711361 16630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9597e28988c41b88feef31ee80f0407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:27.722978 16714 tablet_bootstrap.cc:492] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: Bootstrap starting.
I20260926 02:18:27.724083 16715 tablet_bootstrap.cc:492] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: Bootstrap starting.
I20260926 02:18:27.724900 16716 tablet_bootstrap.cc:492] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923: Bootstrap starting.
I20260926 02:18:27.728893 16714 tablet_bootstrap.cc:654] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:27.728890 16716 tablet_bootstrap.cc:654] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:27.730304 16714 log.cc:824] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:27.730304 16716 log.cc:824] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:27.732486 16715 tablet_bootstrap.cc:654] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:27.733666 16716 tablet_bootstrap.cc:492] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923: No bootstrap required, opened a new log
I20260926 02:18:27.733847 16714 tablet_bootstrap.cc:492] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: No bootstrap required, opened a new log
I20260926 02:18:27.733871 16715 log.cc:824] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:27.733987 16716 ts_tablet_manager.cc:1397] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260926 02:18:27.734151 16714 ts_tablet_manager.cc:1397] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260926 02:18:27.740582 16715 tablet_bootstrap.cc:492] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: No bootstrap required, opened a new log
I20260926 02:18:27.740923 16715 ts_tablet_manager.cc:1397] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20260926 02:18:27.743170 16714 raft_consensus.cc:348] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776daa1d4878453c8af3d8902067ea7f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34147 } } peers { permanent_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45085 } } peers { permanent_uuid: "4952199e44014a8ab139abc350f52d1e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46731 } }
I20260926 02:18:27.743566 16714 raft_consensus.cc:374] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:27.743691 16714 raft_consensus.cc:729] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4952199e44014a8ab139abc350f52d1e, State: Initialized, Role: FOLLOWER
I20260926 02:18:27.744320 16714 consensus_queue.cc:260] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776daa1d4878453c8af3d8902067ea7f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34147 } } peers { permanent_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45085 } } peers { permanent_uuid: "4952199e44014a8ab139abc350f52d1e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46731 } }
I20260926 02:18:27.745059 16716 raft_consensus.cc:348] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776daa1d4878453c8af3d8902067ea7f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34147 } } peers { permanent_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45085 } } peers { permanent_uuid: "4952199e44014a8ab139abc350f52d1e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46731 } }
I20260926 02:18:27.745460 16695 heartbeater.cc:503] Master 127.28.152.190:41393 was elected leader, sending a full tablet report...
I20260926 02:18:27.745893 16716 raft_consensus.cc:374] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:27.746001 16716 raft_consensus.cc:729] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2b521a277f42b2bc1d4848fc9d7923, State: Initialized, Role: FOLLOWER
I20260926 02:18:27.746197 16714 ts_tablet_manager.cc:1428] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260926 02:18:27.746723 16716 consensus_queue.cc:260] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776daa1d4878453c8af3d8902067ea7f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34147 } } peers { permanent_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45085 } } peers { permanent_uuid: "4952199e44014a8ab139abc350f52d1e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46731 } }
I20260926 02:18:27.750249 16716 ts_tablet_manager.cc:1428] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923: Time spent starting tablet: real 0.016s user 0.011s sys 0.002s
I20260926 02:18:27.751405 16715 raft_consensus.cc:348] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776daa1d4878453c8af3d8902067ea7f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34147 } } peers { permanent_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45085 } } peers { permanent_uuid: "4952199e44014a8ab139abc350f52d1e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46731 } }
I20260926 02:18:27.751854 16715 raft_consensus.cc:374] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:27.751957 16715 raft_consensus.cc:729] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 776daa1d4878453c8af3d8902067ea7f, State: Initialized, Role: FOLLOWER
I20260926 02:18:27.752720 16715 consensus_queue.cc:260] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776daa1d4878453c8af3d8902067ea7f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34147 } } peers { permanent_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45085 } } peers { permanent_uuid: "4952199e44014a8ab139abc350f52d1e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46731 } }
I20260926 02:18:27.753921 16560 heartbeater.cc:503] Master 127.28.152.190:41393 was elected leader, sending a full tablet report...
I20260926 02:18:27.754590 16715 ts_tablet_manager.cc:1428] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
W20260926 02:18:27.801565 16426 tablet.cc:2369] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:18:27.839648 16561 tablet.cc:2369] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:18:27.856958 16696 tablet.cc:2369] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:18:27.913519 16721 raft_consensus.cc:482] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:18:27.913813 16721 raft_consensus.cc:504] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776daa1d4878453c8af3d8902067ea7f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34147 } } peers { permanent_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45085 } } peers { permanent_uuid: "4952199e44014a8ab139abc350f52d1e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46731 } }
I20260926 02:18:27.915715 16721 leader_election.cc:288] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 776daa1d4878453c8af3d8902067ea7f (127.28.152.130:34147), 4952199e44014a8ab139abc350f52d1e (127.28.152.131:46731)
I20260926 02:18:27.923311 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.916709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:51191 (local address 127.28.152.131:46731)
0926 02:18:27.916884 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:27.916890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:27.917377 (+ 487us) server_negotiation.cc:407] Connection header received
0926 02:18:27.917601 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:27.917607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:27.917686 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:27.917775 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:27.918492 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.919525 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.921413 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.922397 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.922554 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.922633 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:27.922928 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 02:18:27.923061 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:18:27.923447 16724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.916191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:46731 (local address 127.28.152.129:51191)
0926 02:18:27.917280 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:27.917295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:27.917468 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:27.917824 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:27.917829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:27.917859 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:27.918327 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.918341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.919656 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.919662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.921248 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.921262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.922596 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.922602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.922769 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.922803 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:18:27.922890 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 02:18:27.923015 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":6,"thread_start_us":790,"threads_started":1}
I20260926 02:18:27.923970 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.916210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:48973 (local address 127.28.152.130:34147)
0926 02:18:27.916442 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:27.916449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:27.917091 (+ 642us) server_negotiation.cc:407] Connection header received
0926 02:18:27.917436 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:27.917442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:27.917527 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:27.917626 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:27.918260 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.919347 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.921903 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.922997 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.923151 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.923221 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:27.923593 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 02:18:27.923727 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:18:27.924624 16723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:27.915927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:34147 (local address 127.28.152.129:48973)
0926 02:18:27.916964 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:27.916988 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:27.917251 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:27.917643 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:27.917647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:27.917671 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:27.918103 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.918114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.919489 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.919495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.921726 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:27.921741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:27.923210 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:27.923216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:27.923388 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:27.923425 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:18:27.924150 (+ 725us) client_negotiation.cc:240] Negotiation successful
0926 02:18:27.924283 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":50,"thread_start_us":684,"threads_started":1}
I20260926 02:18:27.924652 16650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9597e28988c41b88feef31ee80f0407" candidate_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4952199e44014a8ab139abc350f52d1e" is_pre_election: true
I20260926 02:18:27.925212 16650 raft_consensus.cc:2454] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef2b521a277f42b2bc1d4848fc9d7923 in term 0.
I20260926 02:18:27.925617 16515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9597e28988c41b88feef31ee80f0407" candidate_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "776daa1d4878453c8af3d8902067ea7f" is_pre_election: true
I20260926 02:18:27.925987 16315 leader_election.cc:302] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4952199e44014a8ab139abc350f52d1e, ef2b521a277f42b2bc1d4848fc9d7923; no voters:
I20260926 02:18:27.926101 16515 raft_consensus.cc:2454] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef2b521a277f42b2bc1d4848fc9d7923 in term 0.
I20260926 02:18:27.926391 16721 raft_consensus.cc:2781] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:18:27.926551 16721 raft_consensus.cc:482] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:18:27.926661 16721 raft_consensus.cc:3037] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:27.928787 16721 raft_consensus.cc:504] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776daa1d4878453c8af3d8902067ea7f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34147 } } peers { permanent_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45085 } } peers { permanent_uuid: "4952199e44014a8ab139abc350f52d1e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46731 } }
I20260926 02:18:27.929430 16721 leader_election.cc:288] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [CANDIDATE]: Term 1 election: Requested vote from peers 776daa1d4878453c8af3d8902067ea7f (127.28.152.130:34147), 4952199e44014a8ab139abc350f52d1e (127.28.152.131:46731)
I20260926 02:18:27.929855 16515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9597e28988c41b88feef31ee80f0407" candidate_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "776daa1d4878453c8af3d8902067ea7f"
I20260926 02:18:27.929935 16650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9597e28988c41b88feef31ee80f0407" candidate_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4952199e44014a8ab139abc350f52d1e"
I20260926 02:18:27.930040 16515 raft_consensus.cc:3037] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:27.930154 16650 raft_consensus.cc:3037] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:27.932358 16515 raft_consensus.cc:2454] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef2b521a277f42b2bc1d4848fc9d7923 in term 1.
I20260926 02:18:27.932361 16650 raft_consensus.cc:2454] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef2b521a277f42b2bc1d4848fc9d7923 in term 1.
I20260926 02:18:27.932780 16312 leader_election.cc:302] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 776daa1d4878453c8af3d8902067ea7f, ef2b521a277f42b2bc1d4848fc9d7923; no voters:
I20260926 02:18:27.933151 16721 raft_consensus.cc:2781] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:27.933568 16721 raft_consensus.cc:686] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 1 LEADER]: Becoming Leader. State: Replica: ef2b521a277f42b2bc1d4848fc9d7923, State: Running, Role: LEADER
I20260926 02:18:27.934032 16721 consensus_queue.cc:237] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776daa1d4878453c8af3d8902067ea7f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34147 } } peers { permanent_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45085 } } peers { permanent_uuid: "4952199e44014a8ab139abc350f52d1e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46731 } }
I20260926 02:18:27.938874 16235 catalog_manager.cc:5495] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef2b521a277f42b2bc1d4848fc9d7923 (127.28.152.129). New cstate: current_term: 1 leader_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776daa1d4878453c8af3d8902067ea7f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef2b521a277f42b2bc1d4848fc9d7923" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 45085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4952199e44014a8ab139abc350f52d1e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46731 } health_report { overall_health: UNKNOWN } } }
I20260926 02:18:28.008121 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:28.002419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.28.152.129:45085)
0926 02:18:28.002672 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:28.002678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:28.002784 (+ 106us) server_negotiation.cc:407] Connection header received
0926 02:18:28.003016 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:28.003020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:28.003114 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:28.003276 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:18:28.003667 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:28.004422 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:28.005923 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:28.006341 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:28.006452 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:28.006976 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:18:28.007514 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:18:28.007651 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:28.007766 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 02:18:28.007858 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 02:18:28.016660 16650 raft_consensus.cc:1261] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e [term 1 FOLLOWER]: Refusing update from remote peer ef2b521a277f42b2bc1d4848fc9d7923: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:28.016660 16515 raft_consensus.cc:1261] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f [term 1 FOLLOWER]: Refusing update from remote peer ef2b521a277f42b2bc1d4848fc9d7923: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:28.017668 16725 consensus_queue.cc:1035] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4952199e44014a8ab139abc350f52d1e" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 46731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:28.018025 16721 consensus_queue.cc:1035] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "776daa1d4878453c8af3d8902067ea7f" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:28.034605 16729 mvcc.cc:205] Tried to move back new op lower bound from 7333433786425954304 to 7333433786101387264. Current Snapshot: MvccSnapshot[applied={T|T < 7333433786425954304}]
I20260926 02:18:28.043676 16235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:28.043931 16235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:28.047767 16235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86e020ecbfbc4a0b809c8f47e9009a54: Sending DeleteTablet for 3 replicas of tablet d9597e28988c41b88feef31ee80f0407
I20260926 02:18:28.049166 16360 tablet_service.cc:1514] Processing DeleteTablet for tablet d9597e28988c41b88feef31ee80f0407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:28 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:42030
I20260926 02:18:28.049297 16630 tablet_service.cc:1514] Processing DeleteTablet for tablet d9597e28988c41b88feef31ee80f0407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:28 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:57276
I20260926 02:18:28.049296 16495 tablet_service.cc:1514] Processing DeleteTablet for tablet d9597e28988c41b88feef31ee80f0407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:28 UTC) from {username='kudu', principal='kudu/127.28.152.190@KRBTEST.COM'} at 127.0.0.1:60506
I20260926 02:18:28.051031 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 16296
I20260926 02:18:28.060243 16739 tablet_replica.cc:330] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: stopping tablet replica
I20260926 02:18:28.061656 16739 raft_consensus.cc:2229] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:28.063086 16739 raft_consensus.cc:2258] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:28.064083 16737 tablet_replica.cc:330] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: stopping tablet replica
I20260926 02:18:28.064579 16737 raft_consensus.cc:2229] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:28.065160 16737 raft_consensus.cc:2258] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:28.066180 16739 ts_tablet_manager.cc:1905] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:28.067745 16737 ts_tablet_manager.cc:1905] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:28.067965 16738 tablet_replica.cc:330] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923: stopping tablet replica
I20260926 02:18:28.068367 16738 raft_consensus.cc:2229] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 1 LEADER]: Raft consensus shutting down.
Sep 26 02:18:28 dist-test-slave-hllj krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.129@KRBTEST.COM
I20260926 02:18:28.073122 16739 ts_tablet_manager.cc:1918] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:28.073454 16739 log.cc:1196] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/d9597e28988c41b88feef31ee80f0407
I20260926 02:18:28.074038 16739 ts_tablet_manager.cc:1939] T d9597e28988c41b88feef31ee80f0407 P 776daa1d4878453c8af3d8902067ea7f: Deleting consensus metadata
I20260926 02:18:28.075376 16737 ts_tablet_manager.cc:1918] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:28.075608 16737 log.cc:1196] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/d9597e28988c41b88feef31ee80f0407
I20260926 02:18:28.068966 16738 raft_consensus.cc:2258] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:28.076066 16219 catalog_manager.cc:4841] TS 776daa1d4878453c8af3d8902067ea7f (127.28.152.130:34147): tablet d9597e28988c41b88feef31ee80f0407 (table test-table [id=d5e5fbf8eeb64cfb91d0f4e74a30adee]) successfully deleted
I20260926 02:18:28.076145 16737 ts_tablet_manager.cc:1939] T d9597e28988c41b88feef31ee80f0407 P 4952199e44014a8ab139abc350f52d1e: Deleting consensus metadata
I20260926 02:18:28.077975 16222 catalog_manager.cc:4841] TS 4952199e44014a8ab139abc350f52d1e (127.28.152.131:46731): tablet d9597e28988c41b88feef31ee80f0407 (table test-table [id=d5e5fbf8eeb64cfb91d0f4e74a30adee]) successfully deleted
I20260926 02:18:28.078584 16738 ts_tablet_manager.cc:1905] T d9597e28988c41b88feef31ee80f0407 P ef2b521a277f42b2bc1d4848fc9d7923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:28.080107 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:28.055398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.28.152.129:45085)
0926 02:18:28.055901 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:28.055908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:28.058680 (+ 2772us) server_negotiation.cc:407] Connection header received
0926 02:18:28.058894 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:28.058899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:28.058992 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:28.059103 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:28.061904 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:28.063151 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:28.065038 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:28.065736 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:28.065857 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:28.071529 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:28.071544 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:28.071564 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:28.071619 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:28.075610 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:28.077225 (+ 1615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:28.077234 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:28.077237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:28.077404 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:28.077967 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:28.077972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:28.078000 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:28.078500 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:28.078815 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:28.079438 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 02:18:28.079696 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358}
I20260926 02:18:28.081573 16420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:28.180675 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 16296
W20260926 02:18:28.196178 16222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.152.129:45085 (error 108)
I20260926 02:18:28.196292 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 16431
I20260926 02:18:28.197887 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:28.197417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45085 (local address 127.0.0.1:42068)
0926 02:18:28.197571 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:28.197696 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 02:18:28.198189 16222 catalog_manager.cc:4584] TS ef2b521a277f42b2bc1d4848fc9d7923 (127.28.152.129:45085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9597e28988c41b88feef31ee80f0407: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45085: connect: Connection refused (error 111)
Sep 26 02:18:28 dist-test-slave-hllj krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.130@KRBTEST.COM
I20260926 02:18:28.211287 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:28.197146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.28.152.130:34147)
0926 02:18:28.197295 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:28.197300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:28.197617 (+ 317us) server_negotiation.cc:407] Connection header received
0926 02:18:28.197760 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:28.197765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:28.197858 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:28.197970 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:28.199314 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:28.200066 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:28.201289 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:28.201674 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:28.201768 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:28.205943 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:28.205960 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:28.205975 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:28.206013 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:28.208747 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:28.209490 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:28.209498 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:28.209501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:28.209625 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:28.209914 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:28.209917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:28.209919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:28.210303 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:28.210543 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:28.210795 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 02:18:28.210973 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 02:18:28.211968 16555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:28.251641 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:28.251167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45085 (local address 127.0.0.1:42078)
0926 02:18:28.251385 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:28.251489 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 02:18:28.252728 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:28.252319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45085 (local address 127.0.0.1:42084)
0926 02:18:28.252476 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:28.252599 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 02:18:28.301290 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 16431
I20260926 02:18:28.313441 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:28.312799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45085 (local address 127.0.0.1:42100)
0926 02:18:28.313077 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:28.313228 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 02:18:28.314604 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:28.314219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45085 (local address 127.0.0.1:42108)
0926 02:18:28.314381 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:28.314469 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 02:18:28.316402 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 16566
Sep 26 02:18:28 dist-test-slave-hllj krb5kdc[16190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.131@KRBTEST.COM
I20260926 02:18:28.333593 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:28.317121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.28.152.131:46731)
0926 02:18:28.317371 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:28.317377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:28.317398 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:18:28.317501 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:28.317506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:28.317580 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:28.317662 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:28.319055 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:28.319774 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:28.321423 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:28.322041 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:28.322183 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:28.327709 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:28.327727 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:28.327745 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:28.327791 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:28.330617 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:28.331412 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:28.331421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:28.331424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:28.331583 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:28.331969 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:28.331973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:28.331975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:28.332367 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:28.332645 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:28.332998 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 02:18:28.333257 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:18:28.334196 16690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:28.395195 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:28.394696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45085 (local address 127.0.0.1:42124)
0926 02:18:28.394917 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:28.395011 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 02:18:28.396200 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:28.395832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:45085 (local address 127.0.0.1:42132)
0926 02:18:28.396020 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:28.396090 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.129:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 02:18:28.421654 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 16566
I20260926 02:18:28.436030 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 16203
I20260926 02:18:28.437233 16265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:28.549445 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 16203
2026-09-26T02:18:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4374 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T02:18:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:28Z Disabled control of system clock
I20260926 02:18:28.623514 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:33791
--webserver_interface=127.28.152.190
--webserver_port=0
--builtin_ntp_servers=127.28.152.148:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:33791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:18:28.961812 16749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:28.962154 16749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:28.962240 16749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:28.970088 16749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:28.970225 16749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:28.970269 16749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:28.970319 16749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:28.970363 16749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:28.981016 16749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:33791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:33791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:28.982662 16749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:28.984536 16749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:28.996701 16755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:28.996850 16757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:28.997080 16754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:28.997475 16749 server_base.cc:1034] running on GCE node
I20260926 02:18:28.998484 16749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:28.999994 16749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:29.001269 16749 hybrid_clock.cc:648] HybridClock initialized: now 1790389109001203 us; error 50 us; skew 500 ppm
I20260926 02:18:29.001652 16749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:29.003907 16749 webserver.cc:460] Webserver started at http://127.28.152.190:44367/ using document root <none> and password file <none>
I20260926 02:18:29.004611 16749 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:29.004726 16749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:29.005041 16749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:29.008167 16749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "55cca42fe26e41969cd5dd53be8337d1"
format_stamp: "Formatted at 2026-09-26 02:18:29 on dist-test-slave-hllj"
server_key: "8b1175e5bdb87ec80bec4e231a3578be"
server_key_iv: "79ce3d0130bc843ee2bcca23dd1f2ffc"
server_key_version: "encryptionkey@0"
I20260926 02:18:29.008998 16749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "55cca42fe26e41969cd5dd53be8337d1"
format_stamp: "Formatted at 2026-09-26 02:18:29 on dist-test-slave-hllj"
server_key: "8b1175e5bdb87ec80bec4e231a3578be"
server_key_iv: "79ce3d0130bc843ee2bcca23dd1f2ffc"
server_key_version: "encryptionkey@0"
I20260926 02:18:29.014510 16749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260926 02:18:29.018307 16763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:29.019737 16749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:29.019912 16749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "55cca42fe26e41969cd5dd53be8337d1"
format_stamp: "Formatted at 2026-09-26 02:18:29 on dist-test-slave-hllj"
server_key: "8b1175e5bdb87ec80bec4e231a3578be"
server_key_iv: "79ce3d0130bc843ee2bcca23dd1f2ffc"
server_key_version: "encryptionkey@0"
I20260926 02:18:29.020124 16749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:29.033067 16749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:29.034051 16749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:29.034372 16749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:29.051841 16749 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:33791
I20260926 02:18:29.051862 16815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:33791 every 8 connection(s)
I20260926 02:18:29.053261 16749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:29.057363 16816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:29.058219 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 16749
I20260926 02:18:29.058378 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:29.058807 29282 external_mini_cluster.cc:1428] Setting key a13b5fcf979254e221c66409301f5294
I20260926 02:18:29.071560 16816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1: Bootstrap starting.
I20260926 02:18:29.073400 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:29.060552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.28.152.190:33791)
0926 02:18:29.061601 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:29.061613 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:29.061661 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:18:29.063833 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:29.063860 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:29.064407 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:29.064783 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:29.065453 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:29.066822 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:29.068718 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:29.069501 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:29.069638 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:29.070926 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:29.070935 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:29.070951 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:29.071498 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:29.071586 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:29.071613 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:29.071718 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:29.071972 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 02:18:29.072308 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":375,"threads_started":1}
I20260926 02:18:29.075170 16816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:29.076581 16816 log.cc:824] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:29.079665 16816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1: No bootstrap required, opened a new log
I20260926 02:18:29.087548 16816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cca42fe26e41969cd5dd53be8337d1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33791 } }
I20260926 02:18:29.087955 16816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:29.088029 16816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55cca42fe26e41969cd5dd53be8337d1, State: Initialized, Role: FOLLOWER
I20260926 02:18:29.088572 16816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cca42fe26e41969cd5dd53be8337d1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33791 } }
I20260926 02:18:29.088855 16816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:29.089023 16816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:29.089258 16816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:29.091665 16816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cca42fe26e41969cd5dd53be8337d1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33791 } }
I20260926 02:18:29.092187 16816 leader_election.cc:302] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55cca42fe26e41969cd5dd53be8337d1; no voters:
I20260926 02:18:29.092758 16816 leader_election.cc:288] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:29.092900 16821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:29.093602 16821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [term 1 LEADER]: Becoming Leader. State: Replica: 55cca42fe26e41969cd5dd53be8337d1, State: Running, Role: LEADER
I20260926 02:18:29.094146 16821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cca42fe26e41969cd5dd53be8337d1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33791 } }
I20260926 02:18:29.095124 16816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:29.097061 16822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55cca42fe26e41969cd5dd53be8337d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cca42fe26e41969cd5dd53be8337d1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33791 } } }
I20260926 02:18:29.097637 16822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:29.098613 16823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55cca42fe26e41969cd5dd53be8337d1. Latest consensus state: current_term: 1 leader_uuid: "55cca42fe26e41969cd5dd53be8337d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55cca42fe26e41969cd5dd53be8337d1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 33791 } } }
I20260926 02:18:29.098914 16823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:29.099339 16830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:29.105018 16830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:29.114392 16830 catalog_manager.cc:1347] Generated new cluster ID: 16b65955ac9540a58733dda29f1ea97d
I20260926 02:18:29.114530 16830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:29.131609 16830 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:29.132972 16830 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:29.148594 16830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1: Generated new TSK 0
I20260926 02:18:29.149848 16830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:29.164439 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.129:0
--local_ip_for_outbound_sockets=127.28.152.129
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33791
--builtin_ntp_servers=127.28.152.148:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:18:29.503005 16840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:29.503341 16840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:29.503432 16840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:29.511065 16840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:29.511204 16840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:29.511314 16840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.129
I20260926 02:18:29.520699 16840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.152.129
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.129
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:29.522038 16840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:29.523849 16840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:29.537389 16848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:29.537417 16846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:29.537462 16845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:29.538184 16840 server_base.cc:1034] running on GCE node
I20260926 02:18:29.538767 16840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:29.539746 16840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:29.540967 16840 hybrid_clock.cc:648] HybridClock initialized: now 1790389109540912 us; error 49 us; skew 500 ppm
I20260926 02:18:29.541388 16840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:29.543642 16840 webserver.cc:460] Webserver started at http://127.28.152.129:33007/ using document root <none> and password file <none>
I20260926 02:18:29.544346 16840 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:29.544447 16840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:29.544759 16840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:29.547853 16840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/data/instance:
uuid: "a385ae46e092400f87b2433dfd44aa4d"
format_stamp: "Formatted at 2026-09-26 02:18:29 on dist-test-slave-hllj"
server_key: "61cafbf99d44fe2be0b3611fe0180b6b"
server_key_iv: "1ca961ee2af23278cce278251f0c5905"
server_key_version: "encryptionkey@0"
I20260926 02:18:29.548689 16840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance:
uuid: "a385ae46e092400f87b2433dfd44aa4d"
format_stamp: "Formatted at 2026-09-26 02:18:29 on dist-test-slave-hllj"
server_key: "61cafbf99d44fe2be0b3611fe0180b6b"
server_key_iv: "1ca961ee2af23278cce278251f0c5905"
server_key_version: "encryptionkey@0"
I20260926 02:18:29.554512 16840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 02:18:29.558637 16854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:29.560269 16840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:29.560453 16840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/wal
uuid: "a385ae46e092400f87b2433dfd44aa4d"
format_stamp: "Formatted at 2026-09-26 02:18:29 on dist-test-slave-hllj"
server_key: "61cafbf99d44fe2be0b3611fe0180b6b"
server_key_iv: "1ca961ee2af23278cce278251f0c5905"
server_key_version: "encryptionkey@0"
I20260926 02:18:29.560680 16840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:29.594849 16840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:29.595801 16840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:29.596112 16840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:29.597157 16840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:29.598661 16840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:29.598755 16840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:29.598865 16840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:29.598935 16840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:29.629520 16840 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.129:41963
I20260926 02:18:29.629596 16966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.129:41963 every 8 connection(s)
I20260926 02:18:29.630885 16840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/data/info.pb
I20260926 02:18:29.638923 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 16840
I20260926 02:18:29.639117 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/wal/instance
I20260926 02:18:29.639549 29282 external_mini_cluster.cc:1428] Setting key 4be0d1d3b76ed401ca994b35ca322141
I20260926 02:18:29.642511 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.130:0
--local_ip_for_outbound_sockets=127.28.152.130
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33791
--builtin_ntp_servers=127.28.152.148:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:18:29.644680 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:29.633487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.129:41505 (local address 127.28.152.190:33791)
0926 02:18:29.633956 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:29.633961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:29.635417 (+ 1456us) server_negotiation.cc:407] Connection header received
0926 02:18:29.637431 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:29.637437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:29.637538 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:29.637634 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:29.638889 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:29.639737 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:29.641922 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:29.642339 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:29.642447 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:29.643874 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:29.643879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:29.643882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:29.643899 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:29.643937 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:29.643950 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:29.644019 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:29.644238 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 02:18:29.644342 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260926 02:18:29.645777 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:29.634193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33791 (local address 127.28.152.129:41505)
0926 02:18:29.635223 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:29.635265 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:29.637036 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:29.637939 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:29.637949 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:29.638055 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:29.638707 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:29.638727 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:29.639881 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:29.639885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:29.641786 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:29.641799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:29.642561 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:29.642568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:29.642860 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:29.643537 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:18:29.643539 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:29.643593 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:18:29.643684 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:29.644033 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:29.644046 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:29.644055 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 02:18:29.644225 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 02:18:29.644660 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":293,"threads_started":1}
I20260926 02:18:29.647475 16967 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33791
I20260926 02:18:29.647787 16967 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:29.648674 16967 heartbeater.cc:511] Master 127.28.152.190:33791 requested a full tablet report, sending...
I20260926 02:18:29.650866 16780 ts_manager.cc:194] Registered new tserver with Master: a385ae46e092400f87b2433dfd44aa4d (127.28.152.129:41963)
I20260926 02:18:29.652359 16780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.129:41505
W20260926 02:18:29.980926 16971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:29.981323 16971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:29.981462 16971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:29.989344 16971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:29.989491 16971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:29.989653 16971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.130
I20260926 02:18:29.999843 16971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.152.130
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.130
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:30.001273 16971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:30.003178 16971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:30.016647 16977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:30.016750 16976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:30.016726 16979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:30.017710 16971 server_base.cc:1034] running on GCE node
I20260926 02:18:30.018356 16971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:30.019300 16971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:30.020529 16971 hybrid_clock.cc:648] HybridClock initialized: now 1790389110020478 us; error 35 us; skew 500 ppm
I20260926 02:18:30.020900 16971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:30.023347 16971 webserver.cc:460] Webserver started at http://127.28.152.130:43693/ using document root <none> and password file <none>
I20260926 02:18:30.024066 16971 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:30.024159 16971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:30.024501 16971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:30.027457 16971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/data/instance:
uuid: "cd615044f9444fcdb6a58d03f5f0c23e"
format_stamp: "Formatted at 2026-09-26 02:18:30 on dist-test-slave-hllj"
server_key: "13e7be122780f4413451388d7e270d26"
server_key_iv: "5e940e9256fd2c4b1b73e3ef3c30a888"
server_key_version: "encryptionkey@0"
I20260926 02:18:30.028251 16971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance:
uuid: "cd615044f9444fcdb6a58d03f5f0c23e"
format_stamp: "Formatted at 2026-09-26 02:18:30 on dist-test-slave-hllj"
server_key: "13e7be122780f4413451388d7e270d26"
server_key_iv: "5e940e9256fd2c4b1b73e3ef3c30a888"
server_key_version: "encryptionkey@0"
I20260926 02:18:30.033684 16971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:18:30.037526 16985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:30.039105 16971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:30.039283 16971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/wal
uuid: "cd615044f9444fcdb6a58d03f5f0c23e"
format_stamp: "Formatted at 2026-09-26 02:18:30 on dist-test-slave-hllj"
server_key: "13e7be122780f4413451388d7e270d26"
server_key_iv: "5e940e9256fd2c4b1b73e3ef3c30a888"
server_key_version: "encryptionkey@0"
I20260926 02:18:30.039513 16971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:30.058498 16971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:30.059365 16971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:30.059659 16971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:30.060672 16971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:30.062273 16971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:30.062371 16971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:30.062472 16971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:30.062541 16971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:30.093160 16971 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.130:33727
I20260926 02:18:30.093230 17097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.130:33727 every 8 connection(s)
I20260926 02:18:30.094468 16971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/data/info.pb
I20260926 02:18:30.099089 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 16971
I20260926 02:18:30.099260 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/wal/instance
I20260926 02:18:30.099695 29282 external_mini_cluster.cc:1428] Setting key 39cd94380daade6b1e7b12a7540d270c
I20260926 02:18:30.102360 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.152.131:0
--local_ip_for_outbound_sockets=127.28.152.131
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33791
--builtin_ntp_servers=127.28.152.148:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:18:30.107396 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.097025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.130:54019 (local address 127.28.152.190:33791)
0926 02:18:30.097227 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:30.097231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:30.098704 (+ 1473us) server_negotiation.cc:407] Connection header received
0926 02:18:30.100615 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:30.100621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:30.100705 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:30.100799 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:30.101992 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.102750 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.104357 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.104776 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.104889 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.106575 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:30.106580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:30.106583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:30.106603 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:30.106638 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:30.106650 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:30.106734 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:30.106969 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 02:18:30.107082 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 02:18:30.108558 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.097578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33791 (local address 127.28.152.130:54019)
0926 02:18:30.098528 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:30.098566 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:30.100337 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:30.101050 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:30.101061 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:30.101194 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:30.101824 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.101843 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.102908 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.102912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.104195 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.104209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.105010 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.105019 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.105304 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.106213 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:18:30.106215 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:30.106270 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:18:30.106368 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:30.106742 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:30.106755 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:30.106764 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 02:18:30.106926 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 02:18:30.107432 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":260,"threads_started":1}
I20260926 02:18:30.110088 17098 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33791
I20260926 02:18:30.110421 17098 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:30.111332 17098 heartbeater.cc:511] Master 127.28.152.190:33791 requested a full tablet report, sending...
I20260926 02:18:30.112950 16780 ts_manager.cc:194] Registered new tserver with Master: cd615044f9444fcdb6a58d03f5f0c23e (127.28.152.130:33727)
I20260926 02:18:30.113984 16780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.130:54019
W20260926 02:18:30.434065 17101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:30.434386 17101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:30.434478 17101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:30.441798 17101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:30.441908 17101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:30.442014 17101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.152.131
I20260926 02:18:30.451350 17101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.152.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.152.131
--webserver_port=0
--tserver_master_addrs=127.28.152.190:33791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.152.131
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:30.452627 17101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:30.454466 17101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:30.468362 17106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:30.468321 17107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:30.468385 17109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:30.469349 17101 server_base.cc:1034] running on GCE node
I20260926 02:18:30.469966 17101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:30.470964 17101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:30.472182 17101 hybrid_clock.cc:648] HybridClock initialized: now 1790389110472124 us; error 54 us; skew 500 ppm
I20260926 02:18:30.472507 17101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:18:30.474836 17101 webserver.cc:460] Webserver started at http://127.28.152.131:46221/ using document root <none> and password file <none>
I20260926 02:18:30.475533 17101 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:30.475634 17101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:30.475932 17101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:30.478865 17101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/data/instance:
uuid: "d781cdea0d2a43e3b30b51a187c990c7"
format_stamp: "Formatted at 2026-09-26 02:18:30 on dist-test-slave-hllj"
server_key: "2a40f047c45f04cf4c68c55d193afcc7"
server_key_iv: "55be4fa3bce634273bf64f6f42d1e34a"
server_key_version: "encryptionkey@0"
I20260926 02:18:30.479624 17101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance:
uuid: "d781cdea0d2a43e3b30b51a187c990c7"
format_stamp: "Formatted at 2026-09-26 02:18:30 on dist-test-slave-hllj"
server_key: "2a40f047c45f04cf4c68c55d193afcc7"
server_key_iv: "55be4fa3bce634273bf64f6f42d1e34a"
server_key_version: "encryptionkey@0"
I20260926 02:18:30.484813 17101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:18:30.488374 17115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:30.489835 17101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:18:30.489991 17101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/wal
uuid: "d781cdea0d2a43e3b30b51a187c990c7"
format_stamp: "Formatted at 2026-09-26 02:18:30 on dist-test-slave-hllj"
server_key: "2a40f047c45f04cf4c68c55d193afcc7"
server_key_iv: "55be4fa3bce634273bf64f6f42d1e34a"
server_key_version: "encryptionkey@0"
I20260926 02:18:30.490257 17101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:30.513293 17101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:30.514197 17101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:30.514499 17101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:30.515527 17101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:18:30.517251 17101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:18:30.517360 17101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:30.517468 17101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:18:30.517537 17101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:30.548732 17101 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.131:39881
I20260926 02:18:30.548797 17227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.131:39881 every 8 connection(s)
I20260926 02:18:30.550151 17101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/data/info.pb
I20260926 02:18:30.558585 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 17101
I20260926 02:18:30.558759 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/wal/instance
I20260926 02:18:30.559165 29282 external_mini_cluster.cc:1428] Setting key 006ada6dee752ee56642ef773310d6ed
I20260926 02:18:30.563045 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.552658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:48709 (local address 127.28.152.190:33791)
0926 02:18:30.552843 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:30.552848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:30.554304 (+ 1456us) server_negotiation.cc:407] Connection header received
0926 02:18:30.556078 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:30.556083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:30.556158 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:30.556250 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:30.557556 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.558403 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.560222 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.560712 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.560842 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.562315 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:30.562320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:30.562323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:30.562346 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:30.562378 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:30.562391 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:30.562465 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:30.562697 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 02:18:30.562800 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:18:30.564025 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.553252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:33791 (local address 127.28.152.131:48709)
0926 02:18:30.554145 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:30.554180 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:30.555795 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:30.556504 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:30.556513 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:30.556632 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:18:30.557353 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.557375 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.558561 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.558567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.560032 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.560044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.560942 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.560950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.561306 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.562008 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:18:30.562011 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:18:30.562053 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:18:30.562142 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:18:30.562469 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:18:30.562475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:18:30.562482 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:18:30.562616 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 02:18:30.562932 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":256,"threads_started":1}
I20260926 02:18:30.565442 17228 heartbeater.cc:347] Connected to a master server at 127.28.152.190:33791
I20260926 02:18:30.565750 17228 heartbeater.cc:464] Registering TS with master...
I20260926 02:18:30.566485 17228 heartbeater.cc:511] Master 127.28.152.190:33791 requested a full tablet report, sending...
I20260926 02:18:30.567947 16780 ts_manager.cc:194] Registered new tserver with Master: d781cdea0d2a43e3b30b51a187c990c7 (127.28.152.131:39881)
I20260926 02:18:30.568753 16780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.152.131:48709
I20260926 02:18:30.574731 29282 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:18:30.584717 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.578817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.28.152.190:33791)
0926 02:18:30.579003 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:30.579010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:30.579389 (+ 379us) server_negotiation.cc:407] Connection header received
0926 02:18:30.579563 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:30.579566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:30.579646 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:30.579731 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:30.580252 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.580976 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.582420 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.582902 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.583019 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.584039 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:30.584044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:30.584048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:30.584067 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:30.584102 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:30.584115 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:30.584188 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:30.584322 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 02:18:30.584421 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:18:30.588593 16780 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:18:30.595682 16780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:18:30.621922 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.609176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.28.152.130:33727)
0926 02:18:30.614003 (+ 4827us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:30.614013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:30.614046 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:18:30.614152 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:30.614161 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:30.614458 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:30.615341 (+ 883us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:30.615535 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.616902 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.619831 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.620785 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.620943 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.621211 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:30.621426 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 02:18:30.621606 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4669,"thread_start_us":4564,"threads_started":1}
I20260926 02:18:30.621986 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.608426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33727 (local address 127.0.0.1:45716)
0926 02:18:30.609354 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:30.609406 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:30.609619 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:30.614733 (+ 5114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:30.614742 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:30.614784 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:30.615318 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.615335 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.617035 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.617041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.619573 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.619583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.620965 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.620972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.621198 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.621256 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 02:18:30.621369 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 02:18:30.621568 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":130,"thread_start_us":1475,"threads_started":1}
I20260926 02:18:30.622404 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.608494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:41963 (local address 127.0.0.1:58154)
0926 02:18:30.610199 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:30.610218 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:30.610334 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:30.614864 (+ 4530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:30.614869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:30.614900 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:30.615415 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.615427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.616799 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.616804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.619562 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.619577 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.621643 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.621649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.621819 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.621854 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:18:30.621942 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 02:18:30.622075 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1561,"mutex_wait_us":56,"thread_start_us":877,"threads_started":1}
I20260926 02:18:30.623281 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.608803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.131:39881 (local address 127.0.0.1:41342)
0926 02:18:30.610874 (+ 2071us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:30.610894 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:30.610988 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:30.615115 (+ 4127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:30.615130 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:30.615153 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:30.615831 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.615842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.617665 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.617671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.619483 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.619496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.622519 (+ 3023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.622528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.622693 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.622724 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:18:30.622798 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 02:18:30.622926 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1966,"mutex_wait_us":58,"thread_start_us":1843,"threads_started":1}
I20260926 02:18:30.623596 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.609172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.28.152.129:41963)
0926 02:18:30.614323 (+ 5151us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:30.614332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:30.614357 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:18:30.614446 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:30.614452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:30.614661 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:30.614854 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:30.615582 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.616678 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.619809 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.621158 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.621293 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.621493 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:30.622954 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0926 02:18:30.623126 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5002,"thread_start_us":4885,"threads_started":1}
I20260926 02:18:30.624045 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.609248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.28.152.131:39881)
0926 02:18:30.614499 (+ 5251us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:30.614507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:30.614545 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:18:30.614657 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:30.614664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:30.614891 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:30.615079 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:30.616028 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.617480 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.619679 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.621044 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.621280 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.621492 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:30.623514 (+ 2022us) server_negotiation.cc:299] Negotiation successful
0926 02:18:30.623702 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5082,"thread_start_us":4087,"threads_started":1}
I20260926 02:18:30.626832 16902 tablet_service.cc:1467] Processing CreateTablet for tablet dba558b3d5cb40e3b90f828032921a28 (DEFAULT_TABLE table=test-table [id=afcca4de7180433192be498da39c6925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:30.626631 17033 tablet_service.cc:1467] Processing CreateTablet for tablet dba558b3d5cb40e3b90f828032921a28 (DEFAULT_TABLE table=test-table [id=afcca4de7180433192be498da39c6925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:30.627550 17163 tablet_service.cc:1467] Processing CreateTablet for tablet dba558b3d5cb40e3b90f828032921a28 (DEFAULT_TABLE table=test-table [id=afcca4de7180433192be498da39c6925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:18:30.628099 17033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba558b3d5cb40e3b90f828032921a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:30.628448 16902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba558b3d5cb40e3b90f828032921a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:30.633119 17163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba558b3d5cb40e3b90f828032921a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:30.645916 17247 tablet_bootstrap.cc:492] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: Bootstrap starting.
I20260926 02:18:30.648972 17246 tablet_bootstrap.cc:492] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: Bootstrap starting.
I20260926 02:18:30.652093 17248 tablet_bootstrap.cc:492] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: Bootstrap starting.
I20260926 02:18:30.652346 17247 tablet_bootstrap.cc:654] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:30.653884 17247 log.cc:824] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:30.655560 17248 tablet_bootstrap.cc:654] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:30.655567 17246 tablet_bootstrap.cc:654] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:30.656796 17248 log.cc:824] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:30.657755 17247 tablet_bootstrap.cc:492] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: No bootstrap required, opened a new log
I20260926 02:18:30.657891 16967 heartbeater.cc:503] Master 127.28.152.190:33791 was elected leader, sending a full tablet report...
I20260926 02:18:30.658033 17247 ts_tablet_manager.cc:1397] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260926 02:18:30.659541 17246 log.cc:824] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:30.659948 17248 tablet_bootstrap.cc:492] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: No bootstrap required, opened a new log
I20260926 02:18:30.660207 17248 ts_tablet_manager.cc:1397] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 02:18:30.667532 17246 tablet_bootstrap.cc:492] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: No bootstrap required, opened a new log
I20260926 02:18:30.667932 17246 ts_tablet_manager.cc:1397] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20260926 02:18:30.669634 17247 raft_consensus.cc:348] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33727 } } peers { permanent_uuid: "a385ae46e092400f87b2433dfd44aa4d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 41963 } } peers { permanent_uuid: "d781cdea0d2a43e3b30b51a187c990c7" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 39881 } }
I20260926 02:18:30.670130 17247 raft_consensus.cc:374] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:30.670223 17247 raft_consensus.cc:729] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a385ae46e092400f87b2433dfd44aa4d, State: Initialized, Role: FOLLOWER
I20260926 02:18:30.670897 17247 consensus_queue.cc:260] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33727 } } peers { permanent_uuid: "a385ae46e092400f87b2433dfd44aa4d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 41963 } } peers { permanent_uuid: "d781cdea0d2a43e3b30b51a187c990c7" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 39881 } }
I20260926 02:18:30.671744 17248 raft_consensus.cc:348] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33727 } } peers { permanent_uuid: "a385ae46e092400f87b2433dfd44aa4d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 41963 } } peers { permanent_uuid: "d781cdea0d2a43e3b30b51a187c990c7" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 39881 } }
I20260926 02:18:30.672207 17248 raft_consensus.cc:374] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:30.672310 17248 raft_consensus.cc:729] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d781cdea0d2a43e3b30b51a187c990c7, State: Initialized, Role: FOLLOWER
I20260926 02:18:30.672940 17248 consensus_queue.cc:260] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33727 } } peers { permanent_uuid: "a385ae46e092400f87b2433dfd44aa4d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 41963 } } peers { permanent_uuid: "d781cdea0d2a43e3b30b51a187c990c7" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 39881 } }
I20260926 02:18:30.674045 17228 heartbeater.cc:503] Master 127.28.152.190:33791 was elected leader, sending a full tablet report...
I20260926 02:18:30.676695 17248 ts_tablet_manager.cc:1428] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260926 02:18:30.680410 17246 raft_consensus.cc:348] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33727 } } peers { permanent_uuid: "a385ae46e092400f87b2433dfd44aa4d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 41963 } } peers { permanent_uuid: "d781cdea0d2a43e3b30b51a187c990c7" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 39881 } }
I20260926 02:18:30.680995 17246 raft_consensus.cc:374] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:30.681167 17246 raft_consensus.cc:729] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd615044f9444fcdb6a58d03f5f0c23e, State: Initialized, Role: FOLLOWER
I20260926 02:18:30.681874 17246 consensus_queue.cc:260] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33727 } } peers { permanent_uuid: "a385ae46e092400f87b2433dfd44aa4d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 41963 } } peers { permanent_uuid: "d781cdea0d2a43e3b30b51a187c990c7" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 39881 } }
I20260926 02:18:30.683027 17098 heartbeater.cc:503] Master 127.28.152.190:33791 was elected leader, sending a full tablet report...
I20260926 02:18:30.683885 17247 ts_tablet_manager.cc:1428] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: Time spent starting tablet: real 0.026s user 0.010s sys 0.015s
I20260926 02:18:30.683889 17246 ts_tablet_manager.cc:1428] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: Time spent starting tablet: real 0.016s user 0.006s sys 0.007s
W20260926 02:18:30.801805 17229 tablet.cc:2369] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:18:30.846491 17099 tablet.cc:2369] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:18:30.883332 16968 tablet.cc:2369] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:18:30.969307 17253 raft_consensus.cc:482] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:18:30.969669 17253 raft_consensus.cc:504] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33727 } } peers { permanent_uuid: "a385ae46e092400f87b2433dfd44aa4d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 41963 } } peers { permanent_uuid: "d781cdea0d2a43e3b30b51a187c990c7" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 39881 } }
I20260926 02:18:30.971412 17253 leader_election.cc:288] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd615044f9444fcdb6a58d03f5f0c23e (127.28.152.130:33727), a385ae46e092400f87b2433dfd44aa4d (127.28.152.129:41963)
I20260926 02:18:30.978605 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.971810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:41141 (local address 127.28.152.130:33727)
0926 02:18:30.972010 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:30.972015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:30.972073 (+ 58us) server_negotiation.cc:407] Connection header received
0926 02:18:30.972364 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:30.972370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:30.972464 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:30.972560 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:30.973106 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.974078 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.976578 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.977682 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.977826 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.977899 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:30.978202 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 02:18:30.978337 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:18:30.978879 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.972307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.152.131:37349 (local address 127.28.152.129:41963)
0926 02:18:30.972496 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:30.972501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:30.972724 (+ 223us) server_negotiation.cc:407] Connection header received
0926 02:18:30.972882 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:30.972886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:30.972946 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:30.973027 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:18:30.973588 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.974477 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.976936 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.978022 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.978157 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.978223 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:30.978514 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 02:18:30.978638 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:18:30.978962 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.971888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.129:41963 (local address 127.28.152.131:37349)
0926 02:18:30.972634 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:30.972650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:30.972764 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:30.973076 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:30.973080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:30.973101 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:30.973418 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.973427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.974626 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.974631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.976758 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.976774 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.978225 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.978233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.978367 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.978390 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:18:30.978467 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 02:18:30.978597 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":98,"thread_start_us":399,"threads_started":1}
I20260926 02:18:30.979269 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:30.971701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.130:33727 (local address 127.28.152.131:41141)
0926 02:18:30.971986 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 02:18:30.972006 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:18:30.972155 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:18:30.972745 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:18:30.972750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:18:30.972771 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:18:30.972948 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.972957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.974238 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.974242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.976407 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:18:30.976425 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:18:30.977831 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:18:30.977837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:18:30.978011 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:30.978036 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:18:30.978806 (+ 770us) client_negotiation.cc:240] Negotiation successful
0926 02:18:30.978959 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":43}
I20260926 02:18:30.979887 16922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba558b3d5cb40e3b90f828032921a28" candidate_uuid: "d781cdea0d2a43e3b30b51a187c990c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a385ae46e092400f87b2433dfd44aa4d" is_pre_election: true
I20260926 02:18:30.980101 17053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba558b3d5cb40e3b90f828032921a28" candidate_uuid: "d781cdea0d2a43e3b30b51a187c990c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" is_pre_election: true
I20260926 02:18:30.980401 16922 raft_consensus.cc:2454] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d781cdea0d2a43e3b30b51a187c990c7 in term 0.
I20260926 02:18:30.980623 17053 raft_consensus.cc:2454] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d781cdea0d2a43e3b30b51a187c990c7 in term 0.
I20260926 02:18:30.981017 17116 leader_election.cc:302] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a385ae46e092400f87b2433dfd44aa4d, d781cdea0d2a43e3b30b51a187c990c7; no voters:
I20260926 02:18:30.981439 17253 raft_consensus.cc:2781] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:18:30.981592 17253 raft_consensus.cc:482] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:18:30.981707 17253 raft_consensus.cc:3037] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:30.983845 17253 raft_consensus.cc:504] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33727 } } peers { permanent_uuid: "a385ae46e092400f87b2433dfd44aa4d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 41963 } } peers { permanent_uuid: "d781cdea0d2a43e3b30b51a187c990c7" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 39881 } }
I20260926 02:18:30.984414 17253 leader_election.cc:288] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [CANDIDATE]: Term 1 election: Requested vote from peers cd615044f9444fcdb6a58d03f5f0c23e (127.28.152.130:33727), a385ae46e092400f87b2433dfd44aa4d (127.28.152.129:41963)
I20260926 02:18:30.984942 17053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba558b3d5cb40e3b90f828032921a28" candidate_uuid: "d781cdea0d2a43e3b30b51a187c990c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd615044f9444fcdb6a58d03f5f0c23e"
I20260926 02:18:30.984943 16922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dba558b3d5cb40e3b90f828032921a28" candidate_uuid: "d781cdea0d2a43e3b30b51a187c990c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a385ae46e092400f87b2433dfd44aa4d"
I20260926 02:18:30.985203 16922 raft_consensus.cc:3037] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:30.985190 17053 raft_consensus.cc:3037] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:30.987351 17053 raft_consensus.cc:2454] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d781cdea0d2a43e3b30b51a187c990c7 in term 1.
I20260926 02:18:30.987388 16922 raft_consensus.cc:2454] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d781cdea0d2a43e3b30b51a187c990c7 in term 1.
I20260926 02:18:30.987816 17117 leader_election.cc:302] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd615044f9444fcdb6a58d03f5f0c23e, d781cdea0d2a43e3b30b51a187c990c7; no voters:
I20260926 02:18:30.988129 17253 raft_consensus.cc:2781] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:30.988500 17253 raft_consensus.cc:686] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 1 LEADER]: Becoming Leader. State: Replica: d781cdea0d2a43e3b30b51a187c990c7, State: Running, Role: LEADER
I20260926 02:18:30.988986 17253 consensus_queue.cc:237] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33727 } } peers { permanent_uuid: "a385ae46e092400f87b2433dfd44aa4d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 41963 } } peers { permanent_uuid: "d781cdea0d2a43e3b30b51a187c990c7" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 39881 } }
I20260926 02:18:30.993742 16779 catalog_manager.cc:5495] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d781cdea0d2a43e3b30b51a187c990c7 (127.28.152.131). New cstate: current_term: 1 leader_uuid: "d781cdea0d2a43e3b30b51a187c990c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a385ae46e092400f87b2433dfd44aa4d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 41963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d781cdea0d2a43e3b30b51a187c990c7" member_type: VOTER last_known_addr { host: "127.28.152.131" port: 39881 } health_report { overall_health: HEALTHY } } }
I20260926 02:18:31.078563 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:31.072911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.28.152.131:39881)
0926 02:18:31.073151 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:31.073157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:31.073180 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:18:31.073402 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:31.073406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:31.073485 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:31.073645 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:18:31.073993 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:31.074735 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:31.076188 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:31.076646 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:31.076757 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:31.077329 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:18:31.077977 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:18:31.078117 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:31.078227 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 02:18:31.078312 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:18:31.087370 17053 raft_consensus.cc:1261] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e [term 1 FOLLOWER]: Refusing update from remote peer d781cdea0d2a43e3b30b51a187c990c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:31.087399 16922 raft_consensus.cc:1261] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d [term 1 FOLLOWER]: Refusing update from remote peer d781cdea0d2a43e3b30b51a187c990c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:18:31.088308 17256 consensus_queue.cc:1035] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd615044f9444fcdb6a58d03f5f0c23e" member_type: VOTER last_known_addr { host: "127.28.152.130" port: 33727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:31.088641 17253 consensus_queue.cc:1035] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a385ae46e092400f87b2433dfd44aa4d" member_type: VOTER last_known_addr { host: "127.28.152.129" port: 41963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:18:31.105888 17261 mvcc.cc:205] Tried to move back new op lower bound from 7333433799003533312 to 7333433798614913024. Current Snapshot: MvccSnapshot[applied={T|T < 7333433799003533312}]
I20260926 02:18:31.123992 16780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:31.124313 16780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:18:31.128156 16780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55cca42fe26e41969cd5dd53be8337d1: Sending DeleteTablet for 3 replicas of tablet dba558b3d5cb40e3b90f828032921a28
I20260926 02:18:31.129444 16902 tablet_service.cc:1514] Processing DeleteTablet for tablet dba558b3d5cb40e3b90f828032921a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:31 UTC) from {username='slave'} at 127.0.0.1:58154
I20260926 02:18:31.129562 17033 tablet_service.cc:1514] Processing DeleteTablet for tablet dba558b3d5cb40e3b90f828032921a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:31 UTC) from {username='slave'} at 127.0.0.1:45716
I20260926 02:18:31.130348 17269 tablet_replica.cc:330] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: stopping tablet replica
I20260926 02:18:31.130409 17163 tablet_service.cc:1514] Processing DeleteTablet for tablet dba558b3d5cb40e3b90f828032921a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:18:31 UTC) from {username='slave'} at 127.0.0.1:41342
I20260926 02:18:31.130947 17269 raft_consensus.cc:2229] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:31.131587 17269 raft_consensus.cc:2258] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:31.132354 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 16840
I20260926 02:18:31.134176 17269 ts_tablet_manager.cc:1905] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:31.140622 17271 tablet_replica.cc:330] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: stopping tablet replica
I20260926 02:18:31.141255 17271 raft_consensus.cc:2229] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:18:31.141988 17271 raft_consensus.cc:2258] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:31.142346 17269 ts_tablet_manager.cc:1918] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:31.142581 17269 log.cc:1196] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-0/wal/wals/dba558b3d5cb40e3b90f828032921a28
I20260926 02:18:31.143229 17269 ts_tablet_manager.cc:1939] T dba558b3d5cb40e3b90f828032921a28 P a385ae46e092400f87b2433dfd44aa4d: Deleting consensus metadata
I20260926 02:18:31.144132 17270 tablet_replica.cc:330] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: stopping tablet replica
I20260926 02:18:31.144613 17270 raft_consensus.cc:2229] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:18:31.145166 17270 raft_consensus.cc:2258] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:18:31.145691 16764 catalog_manager.cc:4841] TS a385ae46e092400f87b2433dfd44aa4d (127.28.152.129:41963): tablet dba558b3d5cb40e3b90f828032921a28 (table test-table [id=afcca4de7180433192be498da39c6925]) successfully deleted
I20260926 02:18:31.146302 17271 ts_tablet_manager.cc:1905] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:31.149008 17270 ts_tablet_manager.cc:1905] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:18:31.151763 17271 ts_tablet_manager.cc:1918] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:31.151934 17271 log.cc:1196] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-2/wal/wals/dba558b3d5cb40e3b90f828032921a28
I20260926 02:18:31.152498 17271 ts_tablet_manager.cc:1939] T dba558b3d5cb40e3b90f828032921a28 P d781cdea0d2a43e3b30b51a187c990c7: Deleting consensus metadata
I20260926 02:18:31.154417 16767 catalog_manager.cc:4841] TS d781cdea0d2a43e3b30b51a187c990c7 (127.28.152.131:39881): tablet dba558b3d5cb40e3b90f828032921a28 (table test-table [id=afcca4de7180433192be498da39c6925]) successfully deleted
I20260926 02:18:31.156126 17270 ts_tablet_manager.cc:1918] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:18:31.156311 17270 log.cc:1196] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: Deleting WAL directory at /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790388791636863-29282-0/minicluster-data/ts-1/wal/wals/dba558b3d5cb40e3b90f828032921a28
I20260926 02:18:31.156872 17270 ts_tablet_manager.cc:1939] T dba558b3d5cb40e3b90f828032921a28 P cd615044f9444fcdb6a58d03f5f0c23e: Deleting consensus metadata
I20260926 02:18:31.158946 16765 catalog_manager.cc:4841] TS cd615044f9444fcdb6a58d03f5f0c23e (127.28.152.130:33727): tablet dba558b3d5cb40e3b90f828032921a28 (table test-table [id=afcca4de7180433192be498da39c6925]) successfully deleted
I20260926 02:18:31.167205 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:31.133823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.28.152.129:41963)
0926 02:18:31.134001 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:31.134007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:31.153591 (+ 19584us) server_negotiation.cc:407] Connection header received
0926 02:18:31.153898 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:31.153904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:31.154030 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:31.154114 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:31.154902 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:31.155630 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:31.157072 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:31.157525 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:31.157643 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:31.165683 (+ 8040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:31.165698 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:31.165715 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:31.166344 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:31.166405 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:31.166432 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:31.166582 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:31.166738 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 02:18:31.166868 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:18:31.167934 16962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:31.257261 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 16840
I20260926 02:18:31.271618 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 16971
I20260926 02:18:31.278432 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:31.272353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.28.152.130:33727)
0926 02:18:31.272558 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:31.272565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:31.272587 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:18:31.272785 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:31.272792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:31.272879 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:31.272962 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:31.273391 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:31.274133 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:31.275416 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:31.275850 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:31.275958 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:31.277018 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:31.277028 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:31.277043 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:31.277669 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:31.277719 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:31.277739 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:31.277835 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:31.277982 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 02:18:31.278101 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 02:18:31.279038 17093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:31.396914 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 16971
I20260926 02:18:31.410728 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17101
I20260926 02:18:31.418761 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:31.411544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.28.152.131:39881)
0926 02:18:31.411720 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:31.411724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:31.411743 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:18:31.411948 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:31.411952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:31.412047 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:31.412188 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:31.412668 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:31.413605 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:31.415540 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:31.415980 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:31.416079 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:31.417297 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:31.417306 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:31.417322 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:31.417954 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:31.418005 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:31.418028 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:31.418128 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:31.418317 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 02:18:31.418408 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:18:31.419430 17223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:31.539455 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 17101
I20260926 02:18:31.556026 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 16749
I20260926 02:18:31.557168 16810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:31.665966 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 16749
2026-09-26T02:18:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3115 ms)
[----------] 6 tests from EncryptionPolicyTest (22288 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:31 dist-test-slave-hllj krb5kdc[17279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:31 dist-test-slave-hllj krb5kdc[17279](info): set up 2 sockets
Sep 26 02:18:31 dist-test-slave-hllj krb5kdc[17279](info): commencing operation
krb5kdc: starting...
W20260926 02:18:32.718920 29282 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:32 dist-test-slave-hllj krb5kdc[17279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:32.970618 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:33.302172 17292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:33.302479 17292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:33.302615 17292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:33.309923 17292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:33.310070 17292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:33.310202 17292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:33.310285 17292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:33.310344 17292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:33.320732 17292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:33.322163 17292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:33.323530 17292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:33.335752 17297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:33.335835 17300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:33.335786 17298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:33.336532 17292 server_base.cc:1034] running on GCE node
I20260926 02:18:33.337462 17292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:33.339062 17292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:33.340297 17292 hybrid_clock.cc:648] HybridClock initialized: now 1790389113340241 us; error 48 us; skew 500 ppm
I20260926 02:18:33.340685 17292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:33 dist-test-slave-hllj krb5kdc[17279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:33.344146 17292 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:33.345782 17292 webserver.cc:460] Webserver started at http://127.0.0.1:34539/ using document root <none> and password file <none>
I20260926 02:18:33.346524 17292 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:33.346653 17292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:33.346946 17292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:33.350183 17292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "f42b0ee6c1a447f59885a733d28d239e"
format_stamp: "Formatted at 2026-09-26 02:18:33 on dist-test-slave-hllj"
server_key: "15cbfe53e04353030676d1e41034fcc7"
server_key_iv: "a594b7dbfd0ceaa5ef69e683868e262d"
server_key_version: "encryptionkey@0"
I20260926 02:18:33.351025 17292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "f42b0ee6c1a447f59885a733d28d239e"
format_stamp: "Formatted at 2026-09-26 02:18:33 on dist-test-slave-hllj"
server_key: "15cbfe53e04353030676d1e41034fcc7"
server_key_iv: "a594b7dbfd0ceaa5ef69e683868e262d"
server_key_version: "encryptionkey@0"
I20260926 02:18:33.356771 17292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 02:18:33.360975 17307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:33.362691 17292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:18:33.362907 17292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "f42b0ee6c1a447f59885a733d28d239e"
format_stamp: "Formatted at 2026-09-26 02:18:33 on dist-test-slave-hllj"
server_key: "15cbfe53e04353030676d1e41034fcc7"
server_key_iv: "a594b7dbfd0ceaa5ef69e683868e262d"
server_key_version: "encryptionkey@0"
I20260926 02:18:33.363122 17292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:33.387131 17292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:33.391919 17292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:33.392199 17292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:33.410662 17292 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38153
I20260926 02:18:33.410698 17358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38153 every 8 connection(s)
I20260926 02:18:33.412097 17292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:33.415321 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 17292
I20260926 02:18:33.415509 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:33.416177 29282 external_mini_cluster.cc:1428] Setting key 3fe1d479ca6979292c5cfbce3a1ed6ed
I20260926 02:18:33.416170 17359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:33.431190 17359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e: Bootstrap starting.
Sep 26 02:18:33 dist-test-slave-hllj krb5kdc[17279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:18:33.435199 17359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:33.436748 17359 log.cc:824] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:33.440589 17359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e: No bootstrap required, opened a new log
I20260926 02:18:33.442304 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:33.418263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60195 (local address 127.0.0.1:38153)
0926 02:18:33.419094 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:33.419113 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:33.419169 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:18:33.421455 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:33.421491 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:33.422145 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:33.422670 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:33.424154 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:33.426144 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:33.427581 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:33.428722 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:33.428923 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:33.433025 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:33.433038 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:33.433060 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:33.433113 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:33.437106 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:33.438315 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:33.438325 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:33.438335 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:33.438594 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:33.439053 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:33.439058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:33.439061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:33.439798 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:33.440111 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:33.440507 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 02:18:33.440870 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":345,"threads_started":1}
I20260926 02:18:33.446724 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:33.450425 17359 raft_consensus.cc:348] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b0ee6c1a447f59885a733d28d239e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } }
I20260926 02:18:33.451027 17359 raft_consensus.cc:374] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:33.451155 17359 raft_consensus.cc:729] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42b0ee6c1a447f59885a733d28d239e, State: Initialized, Role: FOLLOWER
I20260926 02:18:33.451999 17359 consensus_queue.cc:260] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b0ee6c1a447f59885a733d28d239e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } }
I20260926 02:18:33.452342 17359 raft_consensus.cc:388] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:33.452529 17359 raft_consensus.cc:482] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:33.452729 17359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:33.456137 17359 raft_consensus.cc:504] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b0ee6c1a447f59885a733d28d239e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } }
I20260926 02:18:33.456808 17359 leader_election.cc:302] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f42b0ee6c1a447f59885a733d28d239e; no voters:
I20260926 02:18:33.457572 17359 leader_election.cc:288] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:33.457690 17369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:33.460042 17359 sys_catalog.cc:564] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:33.463137 17369 raft_consensus.cc:686] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [term 1 LEADER]: Becoming Leader. State: Replica: f42b0ee6c1a447f59885a733d28d239e, State: Running, Role: LEADER
I20260926 02:18:33.464073 17369 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b0ee6c1a447f59885a733d28d239e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } }
I20260926 02:18:33.467440 17370 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f42b0ee6c1a447f59885a733d28d239e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b0ee6c1a447f59885a733d28d239e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } } }
I20260926 02:18:33.467890 17370 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:33.468415 17371 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42b0ee6c1a447f59885a733d28d239e. Latest consensus state: current_term: 1 leader_uuid: "f42b0ee6c1a447f59885a733d28d239e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b0ee6c1a447f59885a733d28d239e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } } }
I20260926 02:18:33.468704 17371 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:33.472206 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:33.454099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46603 (local address 127.0.0.1:38153)
0926 02:18:33.454320 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:33.454327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:33.454825 (+ 498us) server_negotiation.cc:407] Connection header received
0926 02:18:33.455020 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:33.455030 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:33.455101 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:33.455189 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:33.457080 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:33.458160 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:33.459652 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:33.460115 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:33.460926 (+ 811us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:33.463019 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:33.463029 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:33.463033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:33.463084 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:33.466793 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:33.468262 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:33.468267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:33.468270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:33.468440 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:33.469184 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:33.469189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:33.469197 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:33.469692 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:33.471376 (+ 1684us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:33.471569 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 02:18:33.471793 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:18:33.482633 17380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:33.487287 17380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:33.495592 17380 catalog_manager.cc:1347] Generated new cluster ID: 1c5604c5c60b44698e3b947dd3ddbf53
I20260926 02:18:33.495740 17380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:33.524633 17380 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:33.526108 17380 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:33.532182 17380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f42b0ee6c1a447f59885a733d28d239e: Generated new TSK 0
I20260926 02:18:33.533077 17380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:33.538839 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17292
I20260926 02:18:33.540550 17354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:33.633038 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 17292
W20260926 02:18:33.647399 17277 connection.cc:535] client connection to 127.0.0.1:38153 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38153): Connection reset by peer (error 104)
2026-09-26T02:18:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:33 dist-test-slave-hllj krb5kdc[17392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:33 dist-test-slave-hllj krb5kdc[17392](info): set up 2 sockets
Sep 26 02:18:33 dist-test-slave-hllj krb5kdc[17392](info): commencing operation
krb5kdc: starting...
W20260926 02:18:34.662815 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:34 dist-test-slave-hllj krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:34.914057 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:35.244812 17405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:35.245178 17405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:35.245329 17405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:35.252782 17405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:35.252948 17405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:35.253034 17405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:35.253121 17405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:35.253209 17405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:35.264151 17405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:35.265668 17405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:35.267015 17405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:35.278810 17411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:35.278887 17413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:35.278851 17410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:35.279624 17405 server_base.cc:1034] running on GCE node
I20260926 02:18:35.280426 17405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:35.281919 17405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:35.283136 17405 hybrid_clock.cc:648] HybridClock initialized: now 1790389115283078 us; error 51 us; skew 500 ppm
I20260926 02:18:35.283491 17405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:35 dist-test-slave-hllj krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:35.286823 17405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:35.288388 17405 webserver.cc:460] Webserver started at http://127.0.0.1:41377/ using document root <none> and password file <none>
I20260926 02:18:35.289091 17405 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:35.289251 17405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:35.289559 17405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:35.292641 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "3aaea9f50a0e4a68b7fe7fe3bd3abe42"
format_stamp: "Formatted at 2026-09-26 02:18:35 on dist-test-slave-hllj"
server_key: "477eeb0f80752b82b9423c7da1798a0f"
server_key_iv: "99414cec18f0522aa92824f31d69d9f1"
server_key_version: "encryptionkey@0"
I20260926 02:18:35.293466 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "3aaea9f50a0e4a68b7fe7fe3bd3abe42"
format_stamp: "Formatted at 2026-09-26 02:18:35 on dist-test-slave-hllj"
server_key: "477eeb0f80752b82b9423c7da1798a0f"
server_key_iv: "99414cec18f0522aa92824f31d69d9f1"
server_key_version: "encryptionkey@0"
I20260926 02:18:35.298703 17405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 02:18:35.302271 17420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:35.303627 17405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:18:35.303791 17405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "3aaea9f50a0e4a68b7fe7fe3bd3abe42"
format_stamp: "Formatted at 2026-09-26 02:18:35 on dist-test-slave-hllj"
server_key: "477eeb0f80752b82b9423c7da1798a0f"
server_key_iv: "99414cec18f0522aa92824f31d69d9f1"
server_key_version: "encryptionkey@0"
I20260926 02:18:35.303984 17405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:35.319433 17405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:35.323757 17405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:35.324038 17405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:35.342316 17405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46859
I20260926 02:18:35.342352 17471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46859 every 8 connection(s)
I20260926 02:18:35.343798 17405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:35.347880 17472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:35.348342 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 17405
I20260926 02:18:35.348533 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:35.349100 29282 external_mini_cluster.cc:1428] Setting key 6d54c125aa5f01a8936816578b53a025
I20260926 02:18:35.361181 17472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42: Bootstrap starting.
I20260926 02:18:35.363411 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:35.351185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60849 (local address 127.0.0.1:46859)
0926 02:18:35.352051 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:35.352103 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:35.352147 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:18:35.354538 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:35.354568 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:35.355050 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:35.355469 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:35.356028 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:35.357361 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:35.358784 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:35.359514 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:35.359695 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:35.361022 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:35.361030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:35.361046 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:35.361694 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:35.361781 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:35.361806 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:35.361954 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:35.362241 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 02:18:35.362412 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":379,"threads_started":1}
I20260926 02:18:35.364557 17472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:35.365955 17472 log.cc:824] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:35.367650 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:35.369045 17472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42: No bootstrap required, opened a new log
I20260926 02:18:35.377516 17472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaea9f50a0e4a68b7fe7fe3bd3abe42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } }
I20260926 02:18:35.377925 17472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:35.378058 17472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aaea9f50a0e4a68b7fe7fe3bd3abe42, State: Initialized, Role: FOLLOWER
I20260926 02:18:35.378410 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:35.372221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39625 (local address 127.0.0.1:46859)
0926 02:18:35.372378 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:35.372384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:35.372822 (+ 438us) server_negotiation.cc:407] Connection header received
0926 02:18:35.373005 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:35.373009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:35.373080 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:35.373209 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:35.373699 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:35.374505 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:35.375851 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:35.376328 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:35.376438 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:35.377628 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:35.377633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:35.377636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:35.377654 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:35.377708 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:35.377728 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:35.377833 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:35.378041 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 02:18:35.378106 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 02:18:35.378777 17472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaea9f50a0e4a68b7fe7fe3bd3abe42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } }
I20260926 02:18:35.379071 17472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:35.379226 17472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:35.379400 17472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:35.381739 17472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaea9f50a0e4a68b7fe7fe3bd3abe42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } }
I20260926 02:18:35.382242 17472 leader_election.cc:302] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aaea9f50a0e4a68b7fe7fe3bd3abe42; no voters:
I20260926 02:18:35.382763 17472 leader_election.cc:288] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:35.382896 17482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:35.384006 17482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [term 1 LEADER]: Becoming Leader. State: Replica: 3aaea9f50a0e4a68b7fe7fe3bd3abe42, State: Running, Role: LEADER
I20260926 02:18:35.384685 17482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaea9f50a0e4a68b7fe7fe3bd3abe42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } }
I20260926 02:18:35.385118 17472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:35.387753 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aaea9f50a0e4a68b7fe7fe3bd3abe42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaea9f50a0e4a68b7fe7fe3bd3abe42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } } }
I20260926 02:18:35.388119 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:35.388514 17484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aaea9f50a0e4a68b7fe7fe3bd3abe42. Latest consensus state: current_term: 1 leader_uuid: "3aaea9f50a0e4a68b7fe7fe3bd3abe42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aaea9f50a0e4a68b7fe7fe3bd3abe42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46859 } } }
I20260926 02:18:35.388758 17484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:35.390484 17488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:35.396651 17488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:35.408131 17488 catalog_manager.cc:1347] Generated new cluster ID: 8ea670992d62488d8e0defdeb8e03333
I20260926 02:18:35.408258 17488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:35.432060 17488 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:35.433490 17488 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:35.449632 17488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aaea9f50a0e4a68b7fe7fe3bd3abe42: Generated new TSK 0
I20260926 02:18:35.450826 17488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:35.468176 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17405
W20260926 02:18:35.469703 17467 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60849
W20260926 02:18:35.470562 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17405: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:18:35.470769 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 17405
W20260926 02:18:35.484726 17390 connection.cc:535] client connection to 127.0.0.1:46859 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46859): Connection reset by peer (error 104)
2026-09-26T02:18:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1837 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:35 dist-test-slave-hllj krb5kdc[17503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:35 dist-test-slave-hllj krb5kdc[17503](info): set up 2 sockets
Sep 26 02:18:35 dist-test-slave-hllj krb5kdc[17503](info): commencing operation
krb5kdc: starting...
W20260926 02:18:36.498737 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:36 dist-test-slave-hllj krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:36.749289 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:37.084352 17516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:37.084676 17516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:37.084821 17516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:37.092311 17516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:37.092432 17516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:37.092506 17516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:37.092593 17516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:37.092653 17516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:37.102870 17516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:37.104235 17516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:37.105592 17516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:37.117662 17521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:37.117688 17524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:37.117633 17522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:37.118492 17516 server_base.cc:1034] running on GCE node
I20260926 02:18:37.119284 17516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:37.120772 17516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:37.121971 17516 hybrid_clock.cc:648] HybridClock initialized: now 1790389117121907 us; error 56 us; skew 500 ppm
I20260926 02:18:37.122361 17516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:37 dist-test-slave-hllj krb5kdc[17503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:37.129575 17516 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:37.131412 17516 webserver.cc:460] Webserver started at http://127.0.0.1:43797/ using document root <none> and password file <none>
I20260926 02:18:37.132237 17516 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:37.132370 17516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:37.132697 17516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:37.136319 17516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "25627cdddb304d298ae97e7697028259"
format_stamp: "Formatted at 2026-09-26 02:18:37 on dist-test-slave-hllj"
server_key: "5b3f8994fd87011b26be9a8a5d7af7d6"
server_key_iv: "fe07868fc282d99da665f7fe40aebe67"
server_key_version: "encryptionkey@0"
I20260926 02:18:37.137264 17516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "25627cdddb304d298ae97e7697028259"
format_stamp: "Formatted at 2026-09-26 02:18:37 on dist-test-slave-hllj"
server_key: "5b3f8994fd87011b26be9a8a5d7af7d6"
server_key_iv: "fe07868fc282d99da665f7fe40aebe67"
server_key_version: "encryptionkey@0"
I20260926 02:18:37.143067 17516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:18:37.146793 17531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:37.148306 17516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:18:37.148475 17516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "25627cdddb304d298ae97e7697028259"
format_stamp: "Formatted at 2026-09-26 02:18:37 on dist-test-slave-hllj"
server_key: "5b3f8994fd87011b26be9a8a5d7af7d6"
server_key_iv: "fe07868fc282d99da665f7fe40aebe67"
server_key_version: "encryptionkey@0"
I20260926 02:18:37.148710 17516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:37.154799 17516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:37.159204 17516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:37.159520 17516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:37.177605 17516 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42669
I20260926 02:18:37.177657 17582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42669 every 8 connection(s)
I20260926 02:18:37.179136 17516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:37.183259 17583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:37.184154 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 17516
I20260926 02:18:37.184338 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:37.184763 29282 external_mini_cluster.cc:1428] Setting key 7115a3bed7ad2b310c94b0a07750ddfc
I20260926 02:18:37.192714 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:37.186548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54553 (local address 127.0.0.1:42669)
0926 02:18:37.187336 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:37.187348 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:37.187395 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:18:37.189315 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:37.189338 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:37.189722 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:37.190082 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:37.191177 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:37.191184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:37.191197 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:37.191211 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:37.191272 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:37.191295 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:37.191451 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:37.191605 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 02:18:37.191782 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":335,"threads_started":1}
I20260926 02:18:37.196386 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:37.196381 17583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259: Bootstrap starting.
I20260926 02:18:37.200287 17583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:37.201620 17583 log.cc:824] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:37.204234 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:37.201623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46737 (local address 127.0.0.1:42669)
0926 02:18:37.201821 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:37.201828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:37.202167 (+ 339us) server_negotiation.cc:407] Connection header received
0926 02:18:37.202384 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:37.202390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:37.202463 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:37.202590 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:37.203550 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:37.203554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:37.203557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:37.203574 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:37.203615 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:37.203630 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:37.203726 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:37.203842 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 02:18:37.203891 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:18:37.205371 17583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259: No bootstrap required, opened a new log
I20260926 02:18:37.212801 17583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25627cdddb304d298ae97e7697028259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } }
I20260926 02:18:37.213280 17583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:37.213378 17583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25627cdddb304d298ae97e7697028259, State: Initialized, Role: FOLLOWER
I20260926 02:18:37.214020 17583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25627cdddb304d298ae97e7697028259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } }
I20260926 02:18:37.214258 17583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:37.214388 17583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:37.214547 17583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:37.216744 17583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25627cdddb304d298ae97e7697028259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } }
I20260926 02:18:37.217408 17583 leader_election.cc:302] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25627cdddb304d298ae97e7697028259; no voters:
I20260926 02:18:37.218050 17583 leader_election.cc:288] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:37.218195 17593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:37.218815 17593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [term 1 LEADER]: Becoming Leader. State: Replica: 25627cdddb304d298ae97e7697028259, State: Running, Role: LEADER
I20260926 02:18:37.219340 17593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25627cdddb304d298ae97e7697028259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } }
I20260926 02:18:37.220084 17583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:37.222198 17595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25627cdddb304d298ae97e7697028259. Latest consensus state: current_term: 1 leader_uuid: "25627cdddb304d298ae97e7697028259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25627cdddb304d298ae97e7697028259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } } }
I20260926 02:18:37.222193 17594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25627cdddb304d298ae97e7697028259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25627cdddb304d298ae97e7697028259" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } } }
I20260926 02:18:37.222683 17595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:37.222764 17594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:37.225289 17599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:37.229993 17599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:37.238621 17599 catalog_manager.cc:1347] Generated new cluster ID: d482d0490dde4031998dde8c493a5f2b
I20260926 02:18:37.238754 17599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:37.259610 17599 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:37.260102 17599 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:37.268472 17599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25627cdddb304d298ae97e7697028259: Generated new TSK 0
I20260926 02:18:37.269699 17599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:37.275641 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17516
W20260926 02:18:37.276925 17578 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54553
W20260926 02:18:37.277776 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17516: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:18:37.277976 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 17516
2026-09-26T02:18:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1806 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:37 dist-test-slave-hllj krb5kdc[17614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:37 dist-test-slave-hllj krb5kdc[17614](info): set up 2 sockets
Sep 26 02:18:37 dist-test-slave-hllj krb5kdc[17614](info): commencing operation
krb5kdc: starting...
W20260926 02:18:38.298491 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:38 dist-test-slave-hllj krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:38.548796 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:38.881825 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:38.882125 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:38.882264 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:38.889587 17627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:38.889719 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:38.889768 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:38.889868 17627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:38.889925 17627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:38.900048 17627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:38.901503 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:38.902863 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:38.915181 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:38.915210 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:38.915386 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:38.916131 17627 server_base.cc:1034] running on GCE node
I20260926 02:18:38.916989 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:38.918890 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:38.920120 17627 hybrid_clock.cc:648] HybridClock initialized: now 1790389118920078 us; error 38 us; skew 500 ppm
I20260926 02:18:38.920499 17627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:38 dist-test-slave-hllj krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:38.923916 17627 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:38.925639 17627 webserver.cc:460] Webserver started at http://127.0.0.1:46093/ using document root <none> and password file <none>
I20260926 02:18:38.926369 17627 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:38.926496 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:38.926800 17627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:38.930006 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "1ebbe9e0138348f2afe1978d2813920c"
format_stamp: "Formatted at 2026-09-26 02:18:38 on dist-test-slave-hllj"
server_key: "55d82431ec49e01509943783c569d973"
server_key_iv: "31ce2ab2571dcb1627d62b07830d6f6d"
server_key_version: "encryptionkey@0"
I20260926 02:18:38.930814 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "1ebbe9e0138348f2afe1978d2813920c"
format_stamp: "Formatted at 2026-09-26 02:18:38 on dist-test-slave-hllj"
server_key: "55d82431ec49e01509943783c569d973"
server_key_iv: "31ce2ab2571dcb1627d62b07830d6f6d"
server_key_version: "encryptionkey@0"
I20260926 02:18:38.936360 17627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:18:38.940225 17642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:38.941835 17627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:18:38.942041 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "1ebbe9e0138348f2afe1978d2813920c"
format_stamp: "Formatted at 2026-09-26 02:18:38 on dist-test-slave-hllj"
server_key: "55d82431ec49e01509943783c569d973"
server_key_iv: "31ce2ab2571dcb1627d62b07830d6f6d"
server_key_version: "encryptionkey@0"
I20260926 02:18:38.942242 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:38.970172 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:38.974932 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:38.975240 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:38.993558 17627 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46589
I20260926 02:18:38.993623 17693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46589 every 8 connection(s)
I20260926 02:18:38.994943 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:38.998782 17694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:39.004206 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 17627
I20260926 02:18:39.004372 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:39.004885 29282 external_mini_cluster.cc:1428] Setting key 7ff20e1bc663ca3f23be1da9ef43f359
I20260926 02:18:39.010336 17694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c: Bootstrap starting.
I20260926 02:18:39.014168 17694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:39.015522 17694 log.cc:824] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:39.019330 17694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c: No bootstrap required, opened a new log
Sep 26 02:18:39 dist-test-slave-hllj krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:18:39.029668 17694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebbe9e0138348f2afe1978d2813920c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } }
I20260926 02:18:39.030198 17694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:39.030319 17694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ebbe9e0138348f2afe1978d2813920c, State: Initialized, Role: FOLLOWER
I20260926 02:18:39.031064 17694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebbe9e0138348f2afe1978d2813920c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } }
I20260926 02:18:39.031383 17694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:39.031440 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:39.006822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38399 (local address 127.0.0.1:46589)
0926 02:18:39.007720 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:39.007733 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:39.007783 (+ 50us) server_negotiation.cc:407] Connection header received
0926 02:18:39.009607 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:39.009633 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:39.010093 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:39.010441 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:39.012320 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:39.013733 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:39.015584 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:39.016568 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:39.016786 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:39.021643 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:39.021655 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:39.021681 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:39.021740 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:39.026152 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:39.027113 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:39.027124 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:39.027135 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:39.027407 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:39.027850 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:39.027855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:39.027859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:39.028678 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:39.028964 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:39.029432 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 02:18:39.029938 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":403,"threads_started":1}
I20260926 02:18:39.031548 17694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:39.031741 17694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:39.034307 17694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebbe9e0138348f2afe1978d2813920c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } }
I20260926 02:18:39.034924 17694 leader_election.cc:302] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ebbe9e0138348f2afe1978d2813920c; no voters:
I20260926 02:18:39.035573 17694 leader_election.cc:288] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:39.035712 17699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:39.035769 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:39.036342 17699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [term 1 LEADER]: Becoming Leader. State: Replica: 1ebbe9e0138348f2afe1978d2813920c, State: Running, Role: LEADER
I20260926 02:18:39.037057 17699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebbe9e0138348f2afe1978d2813920c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } }
I20260926 02:18:39.038800 17694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:39.040438 17700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ebbe9e0138348f2afe1978d2813920c. Latest consensus state: current_term: 1 leader_uuid: "1ebbe9e0138348f2afe1978d2813920c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebbe9e0138348f2afe1978d2813920c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } } }
I20260926 02:18:39.040855 17700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:39.041324 17701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ebbe9e0138348f2afe1978d2813920c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebbe9e0138348f2afe1978d2813920c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } } }
I20260926 02:18:39.041659 17701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:39.043977 17705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:39.050211 17705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:39.073149 17705 catalog_manager.cc:1347] Generated new cluster ID: b25764daefd14cf0acbb1003bd932324
I20260926 02:18:39.073284 17705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:39.090998 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:39.060282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50313 (local address 127.0.0.1:46589)
0926 02:18:39.060476 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:39.060482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:39.071932 (+ 11450us) server_negotiation.cc:407] Connection header received
0926 02:18:39.072167 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:39.072173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:39.072260 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:39.072374 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:39.074555 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:39.075809 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:39.077876 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:39.078644 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:39.078771 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:39.082828 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:39.082836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:39.082845 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:39.082897 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:39.086903 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:39.087938 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:39.087956 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:39.087960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:39.088159 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:39.088681 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:39.088687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:39.088690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:39.089258 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:39.089521 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:39.090273 (+ 752us) server_negotiation.cc:299] Negotiation successful
0926 02:18:39.090549 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:18:39.105994 17705 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:39.107389 17705 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:39.114262 17705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ebbe9e0138348f2afe1978d2813920c: Generated new TSK 0
I20260926 02:18:39.115164 17705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:39.121577 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17627
I20260926 02:18:39.122809 17689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:39.214774 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 17627
2026-09-26T02:18:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:39 dist-test-slave-hllj krb5kdc[17727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:39 dist-test-slave-hllj krb5kdc[17727](info): set up 2 sockets
Sep 26 02:18:39 dist-test-slave-hllj krb5kdc[17727](info): commencing operation
krb5kdc: starting...
W20260926 02:18:40.238636 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:40 dist-test-slave-hllj krb5kdc[17727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:40.488714 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:40.836615 17740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:40.836961 17740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:40.837095 17740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:40.844726 17740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:40.844882 17740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:40.844955 17740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:40.845037 17740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:40.845094 17740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:40.855316 17740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:40.856698 17740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:40.858127 17740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:40.870147 17748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:40.870209 17745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:40.870188 17746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:40.871117 17740 server_base.cc:1034] running on GCE node
I20260926 02:18:40.871948 17740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:40.873437 17740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:40.874667 17740 hybrid_clock.cc:648] HybridClock initialized: now 1790389120874613 us; error 47 us; skew 500 ppm
I20260926 02:18:40.875036 17740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:40 dist-test-slave-hllj krb5kdc[17727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:40.878206 17740 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:40.879693 17740 webserver.cc:460] Webserver started at http://127.0.0.1:39811/ using document root <none> and password file <none>
I20260926 02:18:40.880376 17740 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:40.880505 17740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:40.880795 17740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:40.884060 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "0153bba8839247f9a221d086d3854503"
format_stamp: "Formatted at 2026-09-26 02:18:40 on dist-test-slave-hllj"
server_key: "900232723a42833e51375796cd352f86"
server_key_iv: "dfb4c328e264485ffd9a3f09c0fd0b57"
server_key_version: "encryptionkey@0"
I20260926 02:18:40.884889 17740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "0153bba8839247f9a221d086d3854503"
format_stamp: "Formatted at 2026-09-26 02:18:40 on dist-test-slave-hllj"
server_key: "900232723a42833e51375796cd352f86"
server_key_iv: "dfb4c328e264485ffd9a3f09c0fd0b57"
server_key_version: "encryptionkey@0"
I20260926 02:18:40.890390 17740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:18:40.894236 17755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:40.895656 17740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:18:40.895817 17740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "0153bba8839247f9a221d086d3854503"
format_stamp: "Formatted at 2026-09-26 02:18:40 on dist-test-slave-hllj"
server_key: "900232723a42833e51375796cd352f86"
server_key_iv: "dfb4c328e264485ffd9a3f09c0fd0b57"
server_key_version: "encryptionkey@0"
I20260926 02:18:40.896006 17740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:40.912904 17740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:40.917630 17740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:40.917908 17740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:40.936266 17740 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45737
I20260926 02:18:40.936287 17806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45737 every 8 connection(s)
I20260926 02:18:40.937815 17740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:40.941865 17807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:40.942986 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 17740
I20260926 02:18:40.943164 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:40.943686 29282 external_mini_cluster.cc:1428] Setting key ba2818581068a9147b1d7dbce71f05ac
I20260926 02:18:40.956772 17807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503: Bootstrap starting.
I20260926 02:18:40.958931 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:40.945778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48649 (local address 127.0.0.1:45737)
0926 02:18:40.946615 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:40.946632 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:40.946688 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:18:40.948604 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:40.948638 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:40.949258 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:40.949792 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:40.950325 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:40.952184 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:40.953553 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:40.954691 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:40.954853 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:40.956014 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:40.956025 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:40.956046 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:40.956778 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:40.956889 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:40.956923 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:40.957038 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:40.957266 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 02:18:40.957557 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":325,"threads_started":1}
I20260926 02:18:40.960572 17807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:40.961781 17807 log.cc:824] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:40.963128 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:40.964841 17807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503: No bootstrap required, opened a new log
I20260926 02:18:40.973920 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:40.967396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41365 (local address 127.0.0.1:45737)
0926 02:18:40.967600 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:40.967607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:40.967994 (+ 387us) server_negotiation.cc:407] Connection header received
0926 02:18:40.968147 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:40.968153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:40.968244 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:40.968353 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:40.968852 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:40.969980 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:40.971299 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:40.972003 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:40.972137 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:40.973171 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:40.973179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:40.973184 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:40.973208 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:40.973264 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:40.973287 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:40.973381 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:40.973509 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 02:18:40.973604 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:18:40.974511 17807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0153bba8839247f9a221d086d3854503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260926 02:18:40.975062 17807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:40.975173 17807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0153bba8839247f9a221d086d3854503, State: Initialized, Role: FOLLOWER
I20260926 02:18:40.975838 17807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0153bba8839247f9a221d086d3854503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260926 02:18:40.976071 17807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:40.976176 17807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:40.976296 17807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:40.978583 17807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0153bba8839247f9a221d086d3854503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260926 02:18:40.979198 17807 leader_election.cc:302] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0153bba8839247f9a221d086d3854503; no voters:
I20260926 02:18:40.979969 17807 leader_election.cc:288] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:40.980099 17817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:40.980765 17817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [term 1 LEADER]: Becoming Leader. State: Replica: 0153bba8839247f9a221d086d3854503, State: Running, Role: LEADER
I20260926 02:18:40.981376 17817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0153bba8839247f9a221d086d3854503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260926 02:18:40.982188 17807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:40.984423 17818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0153bba8839247f9a221d086d3854503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0153bba8839247f9a221d086d3854503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } } }
I20260926 02:18:40.984495 17819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0153bba8839247f9a221d086d3854503. Latest consensus state: current_term: 1 leader_uuid: "0153bba8839247f9a221d086d3854503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0153bba8839247f9a221d086d3854503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } } }
I20260926 02:18:40.984905 17818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:40.984925 17819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:40.993734 17825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:40.998044 17825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:41.006402 17825 catalog_manager.cc:1347] Generated new cluster ID: 2f5722e108854727b4858075fa109fb3
I20260926 02:18:41.006538 17825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:41.032648 17825 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:41.034091 17825 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:41.043228 17825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0153bba8839247f9a221d086d3854503: Generated new TSK 0
I20260926 02:18:41.044298 17825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:41.061793 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17740
W20260926 02:18:41.062951 17802 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48649
W20260926 02:18:41.063687 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17740: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:18:41.063831 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 17740
2026-09-26T02:18:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1847 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:41 dist-test-slave-hllj krb5kdc[17838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:41 dist-test-slave-hllj krb5kdc[17838](info): set up 2 sockets
Sep 26 02:18:41 dist-test-slave-hllj krb5kdc[17838](info): commencing operation
krb5kdc: starting...
W20260926 02:18:42.082803 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:42 dist-test-slave-hllj krb5kdc[17838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:42.336834 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:42.665047 17851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:42.665454 17851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:42.665563 17851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:42.673102 17851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:42.673283 17851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:42.673337 17851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:42.673398 17851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:42.673441 17851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:42.683725 17851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:42.685027 17851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:42.686450 17851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:42.698402 17857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:42.698414 17859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:42.698450 17856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:42.699203 17851 server_base.cc:1034] running on GCE node
I20260926 02:18:42.700037 17851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:42.701575 17851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:42.702867 17851 hybrid_clock.cc:648] HybridClock initialized: now 1790389122702759 us; error 40 us; skew 500 ppm
I20260926 02:18:42.703260 17851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:42 dist-test-slave-hllj krb5kdc[17838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:42.710690 17851 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:42.712395 17851 webserver.cc:460] Webserver started at http://127.0.0.1:45163/ using document root <none> and password file <none>
I20260926 02:18:42.713194 17851 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:42.713327 17851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:42.713650 17851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:42.716745 17851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "6e277202fdbf419ba640c20223669beb"
format_stamp: "Formatted at 2026-09-26 02:18:42 on dist-test-slave-hllj"
server_key: "d0bcbdc974656423b6072d3c39427ef2"
server_key_iv: "b5045768eb0d4172099a3c167b9983d5"
server_key_version: "encryptionkey@0"
I20260926 02:18:42.717555 17851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "6e277202fdbf419ba640c20223669beb"
format_stamp: "Formatted at 2026-09-26 02:18:42 on dist-test-slave-hllj"
server_key: "d0bcbdc974656423b6072d3c39427ef2"
server_key_iv: "b5045768eb0d4172099a3c167b9983d5"
server_key_version: "encryptionkey@0"
I20260926 02:18:42.722946 17851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:18:42.726584 17866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:42.728099 17851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:42.728266 17851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "6e277202fdbf419ba640c20223669beb"
format_stamp: "Formatted at 2026-09-26 02:18:42 on dist-test-slave-hllj"
server_key: "d0bcbdc974656423b6072d3c39427ef2"
server_key_iv: "b5045768eb0d4172099a3c167b9983d5"
server_key_version: "encryptionkey@0"
I20260926 02:18:42.728480 17851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:42.734640 17851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:42.739125 17851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:42.739441 17851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:42.757462 17851 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44581
I20260926 02:18:42.757484 17917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44581 every 8 connection(s)
I20260926 02:18:42.759008 17851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:42.761773 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 17851
I20260926 02:18:42.761968 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:42.762526 29282 external_mini_cluster.cc:1428] Setting key fa9697e35e4f4e099c2d0716136854d8
I20260926 02:18:42.763551 17918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:42.770674 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:42.764379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42769 (local address 127.0.0.1:44581)
0926 02:18:42.765284 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:42.765296 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:42.765340 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:18:42.766924 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:42.766957 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:42.767363 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:42.767731 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:42.768845 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:42.768852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:42.768868 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:42.768891 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:42.769012 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:42.769041 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:42.769265 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:42.769371 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 02:18:42.769575 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":335,"threads_started":1}
I20260926 02:18:42.774555 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:42.777590 17918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb: Bootstrap starting.
I20260926 02:18:42.781868 17918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:42.782467 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:42.780065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53517 (local address 127.0.0.1:44581)
0926 02:18:42.780216 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:42.780238 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:42.780484 (+ 246us) server_negotiation.cc:407] Connection header received
0926 02:18:42.780667 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:42.780670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:42.780724 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:42.780808 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:42.781786 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:42.781790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:42.781793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:42.781809 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:42.781849 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:42.781868 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:42.781976 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:42.782125 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 02:18:42.782176 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 02:18:42.783236 17918 log.cc:824] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:42.786206 17918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb: No bootstrap required, opened a new log
I20260926 02:18:42.795153 17918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e277202fdbf419ba640c20223669beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } }
I20260926 02:18:42.795607 17918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:42.795704 17918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e277202fdbf419ba640c20223669beb, State: Initialized, Role: FOLLOWER
I20260926 02:18:42.796358 17918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e277202fdbf419ba640c20223669beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } }
I20260926 02:18:42.796582 17918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:42.796715 17918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:42.796880 17918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:42.799319 17918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e277202fdbf419ba640c20223669beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } }
I20260926 02:18:42.799849 17918 leader_election.cc:302] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e277202fdbf419ba640c20223669beb; no voters:
I20260926 02:18:42.800472 17918 leader_election.cc:288] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:42.800593 17928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:42.801277 17928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [term 1 LEADER]: Becoming Leader. State: Replica: 6e277202fdbf419ba640c20223669beb, State: Running, Role: LEADER
I20260926 02:18:42.801816 17928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e277202fdbf419ba640c20223669beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } }
I20260926 02:18:42.802541 17918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:42.804342 17929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e277202fdbf419ba640c20223669beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e277202fdbf419ba640c20223669beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } } }
I20260926 02:18:42.804823 17929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:42.804656 17930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e277202fdbf419ba640c20223669beb. Latest consensus state: current_term: 1 leader_uuid: "6e277202fdbf419ba640c20223669beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e277202fdbf419ba640c20223669beb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44581 } } }
I20260926 02:18:42.805048 17930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:42.806545 17933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:42.812049 17933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:42.821679 17933 catalog_manager.cc:1347] Generated new cluster ID: 0c89edf7dff14665b0ce0d46da36d32e
I20260926 02:18:42.821807 17933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:42.883946 17933 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:42.884526 17933 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:42.895385 17933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e277202fdbf419ba640c20223669beb: Generated new TSK 0
I20260926 02:18:42.896600 17933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:42.905179 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17851
W20260926 02:18:42.906693 17913 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42769
W20260926 02:18:42.907482 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17851: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:18:42.907670 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 17851
2026-09-26T02:18:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1843 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:43 dist-test-slave-hllj krb5kdc[17949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:43 dist-test-slave-hllj krb5kdc[17949](info): set up 2 sockets
Sep 26 02:18:43 dist-test-slave-hllj krb5kdc[17949](info): commencing operation
krb5kdc: starting...
W20260926 02:18:43.926834 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:44 dist-test-slave-hllj krb5kdc[17949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:44.184684 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33543
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:44.519191 17962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:44.519495 17962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:44.519629 17962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:44.527002 17962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:44.527134 17962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:44.527210 17962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:44.527294 17962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:44.527349 17962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:44.537992 17962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:44.539371 17962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:44.540796 17962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:44.553241 17967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:44.553270 17970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:44.553220 17968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:44.554035 17962 server_base.cc:1034] running on GCE node
I20260926 02:18:44.554832 17962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:44.556335 17962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:44.557548 17962 hybrid_clock.cc:648] HybridClock initialized: now 1790389124557487 us; error 54 us; skew 500 ppm
I20260926 02:18:44.557964 17962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:44 dist-test-slave-hllj krb5kdc[17949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:44.561319 17962 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:44.562947 17962 webserver.cc:460] Webserver started at http://127.0.0.1:37639/ using document root <none> and password file <none>
I20260926 02:18:44.563647 17962 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:44.563750 17962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:44.564018 17962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:44.567178 17962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "aba14b2cace14fe6b11216211d6ce39a"
format_stamp: "Formatted at 2026-09-26 02:18:44 on dist-test-slave-hllj"
server_key: "49e32670a78df52a04160f3bfaeccf07"
server_key_iv: "cf8b7bdc5a66344d35f37311ac231c69"
server_key_version: "encryptionkey@0"
I20260926 02:18:44.567952 17962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "aba14b2cace14fe6b11216211d6ce39a"
format_stamp: "Formatted at 2026-09-26 02:18:44 on dist-test-slave-hllj"
server_key: "49e32670a78df52a04160f3bfaeccf07"
server_key_iv: "cf8b7bdc5a66344d35f37311ac231c69"
server_key_version: "encryptionkey@0"
I20260926 02:18:44.573535 17962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:18:44.577395 17977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:44.578943 17962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:44.579109 17962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "aba14b2cace14fe6b11216211d6ce39a"
format_stamp: "Formatted at 2026-09-26 02:18:44 on dist-test-slave-hllj"
server_key: "49e32670a78df52a04160f3bfaeccf07"
server_key_iv: "cf8b7bdc5a66344d35f37311ac231c69"
server_key_version: "encryptionkey@0"
I20260926 02:18:44.579303 17962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:44.604291 17962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:44.609037 17962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:44.609318 17962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:44.627058 17962 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33543
I20260926 02:18:44.627086 18028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33543 every 8 connection(s)
I20260926 02:18:44.628472 17962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:44.630086 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 17962
I20260926 02:18:44.630270 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:44.630815 29282 external_mini_cluster.cc:1428] Setting key 63c90c5a8da7df002e3c2511d0c6e52d
I20260926 02:18:44.633275 18029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:18:44 dist-test-slave-hllj krb5kdc[17949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:18:44.646562 18029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a: Bootstrap starting.
I20260926 02:18:44.650869 18029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:44.652163 18029 log.cc:824] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:44.655808 18029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a: No bootstrap required, opened a new log
I20260926 02:18:44.656450 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:44.632720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:57421 (local address 127.0.0.1:33543)
0926 02:18:44.633512 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:44.633524 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:44.633571 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:18:44.635259 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:44.635281 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:44.635725 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:44.636112 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:44.637978 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:44.639415 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:44.641266 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:44.642063 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:44.642234 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:44.647199 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:44.647211 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:44.647232 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:44.647290 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:44.651475 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:44.652393 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:44.652404 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:44.652413 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:44.652636 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:44.653238 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:44.653245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:44.653247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:44.653894 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:44.654214 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:44.654739 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 02:18:44.655100 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":347,"threads_started":1}
I20260926 02:18:44.660435 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:44.663945 18029 raft_consensus.cc:348] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba14b2cace14fe6b11216211d6ce39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } }
I20260926 02:18:44.664363 18029 raft_consensus.cc:374] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:44.664470 18029 raft_consensus.cc:729] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba14b2cace14fe6b11216211d6ce39a, State: Initialized, Role: FOLLOWER
I20260926 02:18:44.665166 18029 consensus_queue.cc:260] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba14b2cace14fe6b11216211d6ce39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } }
I20260926 02:18:44.665422 18029 raft_consensus.cc:388] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:44.665561 18029 raft_consensus.cc:482] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:44.665746 18029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:44.668691 18029 raft_consensus.cc:504] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba14b2cace14fe6b11216211d6ce39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } }
I20260926 02:18:44.669368 18029 leader_election.cc:302] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aba14b2cace14fe6b11216211d6ce39a; no voters:
I20260926 02:18:44.670082 18029 leader_election.cc:288] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:44.670181 18039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:44.672304 18039 raft_consensus.cc:686] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [term 1 LEADER]: Becoming Leader. State: Replica: aba14b2cace14fe6b11216211d6ce39a, State: Running, Role: LEADER
I20260926 02:18:44.672401 18029 sys_catalog.cc:564] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:44.672858 18039 consensus_queue.cc:237] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba14b2cace14fe6b11216211d6ce39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } }
I20260926 02:18:44.675722 18041 sys_catalog.cc:455] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aba14b2cace14fe6b11216211d6ce39a. Latest consensus state: current_term: 1 leader_uuid: "aba14b2cace14fe6b11216211d6ce39a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba14b2cace14fe6b11216211d6ce39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } } }
I20260926 02:18:44.676083 18041 sys_catalog.cc:458] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:44.675733 18040 sys_catalog.cc:455] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aba14b2cace14fe6b11216211d6ce39a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba14b2cace14fe6b11216211d6ce39a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } } }
I20260926 02:18:44.676443 18040 sys_catalog.cc:458] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:44.684213 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:44.664733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:57657 (local address 127.0.0.1:33543)
0926 02:18:44.664920 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:44.664926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:44.665427 (+ 501us) server_negotiation.cc:407] Connection header received
0926 02:18:44.665682 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:44.665688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:44.665786 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:44.665899 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:44.667934 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:44.668701 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:44.670547 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:44.671181 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:44.671273 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:44.674867 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:44.674875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:44.674881 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:44.674927 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:44.679020 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:44.680308 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:44.680315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:44.680320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:44.680569 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:44.681217 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:44.681221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:44.681224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:44.681650 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:44.681996 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:44.683486 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0926 02:18:44.683709 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:18:44.690407 18046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:44.695758 18046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:44.706024 18046 catalog_manager.cc:1347] Generated new cluster ID: 6f119fbe3fe14cada5664e6ef22bfbdf
I20260926 02:18:44.706149 18046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:44.722797 18046 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:44.724786 18046 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:44.736053 18046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aba14b2cace14fe6b11216211d6ce39a: Generated new TSK 0
I20260926 02:18:44.737354 18046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:44.744784 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 17962
I20260926 02:18:44.746033 18024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:44.839303 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 17962
W20260926 02:18:44.853502 17947 connection.cc:535] client connection to 127.0.0.1:33543 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33543): Connection reset by peer (error 104)
2026-09-26T02:18:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:44 dist-test-slave-hllj krb5kdc[18061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:44 dist-test-slave-hllj krb5kdc[18061](info): set up 2 sockets
Sep 26 02:18:44 dist-test-slave-hllj krb5kdc[18061](info): commencing operation
krb5kdc: starting...
W20260926 02:18:45.858642 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:46 dist-test-slave-hllj krb5kdc[18061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:46.120572 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:46.456054 18074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:46.456398 18074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:46.456524 18074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:46.464046 18074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:46.464202 18074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:46.464293 18074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:46.464372 18074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:46.464432 18074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:46.474936 18074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:46.476327 18074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:46.477761 18074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:46.489562 18079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:46.489490 18080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:46.489531 18082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:46.490949 18074 server_base.cc:1034] running on GCE node
I20260926 02:18:46.491895 18074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:46.493430 18074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:46.494661 18074 hybrid_clock.cc:648] HybridClock initialized: now 1790389126494596 us; error 51 us; skew 500 ppm
I20260926 02:18:46.495024 18074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:46 dist-test-slave-hllj krb5kdc[18061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:46.498211 18074 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:46.499683 18074 webserver.cc:460] Webserver started at http://127.0.0.1:32917/ using document root <none> and password file <none>
I20260926 02:18:46.500357 18074 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:46.500461 18074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:46.500725 18074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:46.503896 18074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "6251ad272a3f450bb901d79d24701356"
format_stamp: "Formatted at 2026-09-26 02:18:46 on dist-test-slave-hllj"
server_key: "a5acb71985eb51539ed1e0511318da5b"
server_key_iv: "c3ab331212133b195aa2d0bd1eafe6bc"
server_key_version: "encryptionkey@0"
I20260926 02:18:46.504698 18074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "6251ad272a3f450bb901d79d24701356"
format_stamp: "Formatted at 2026-09-26 02:18:46 on dist-test-slave-hllj"
server_key: "a5acb71985eb51539ed1e0511318da5b"
server_key_iv: "c3ab331212133b195aa2d0bd1eafe6bc"
server_key_version: "encryptionkey@0"
I20260926 02:18:46.510334 18074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260926 02:18:46.514125 18089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:46.515585 18074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:46.515755 18074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "6251ad272a3f450bb901d79d24701356"
format_stamp: "Formatted at 2026-09-26 02:18:46 on dist-test-slave-hllj"
server_key: "a5acb71985eb51539ed1e0511318da5b"
server_key_iv: "c3ab331212133b195aa2d0bd1eafe6bc"
server_key_version: "encryptionkey@0"
I20260926 02:18:46.515946 18074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:46.532956 18074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:46.538020 18074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:46.538328 18074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:46.556558 18074 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35793
I20260926 02:18:46.556579 18140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35793 every 8 connection(s)
I20260926 02:18:46.557984 18074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:46.561971 18141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:46.565584 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 18074
I20260926 02:18:46.565743 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:46.566174 29282 external_mini_cluster.cc:1428] Setting key 8f869d33afc17b79b4fbca7b3932f071
I20260926 02:18:46.574463 18141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356: Bootstrap starting.
I20260926 02:18:46.577843 18141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:46.578991 18141 log.cc:824] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:46.580240 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:46.568146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:37185 (local address 127.0.0.1:35793)
0926 02:18:46.568979 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:46.568992 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:46.569036 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:18:46.570916 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:46.570941 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:46.571493 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:46.571997 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:46.572607 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:46.574010 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:46.575406 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:46.576126 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:46.576295 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:46.577640 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:46.577648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:46.577664 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:46.578382 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:46.578468 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:46.578495 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:46.578639 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:46.578955 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 02:18:46.579133 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":314,"threads_started":1}
I20260926 02:18:46.582027 18141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356: No bootstrap required, opened a new log
I20260926 02:18:46.584379 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:46.590154 18141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6251ad272a3f450bb901d79d24701356" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35793 } }
I20260926 02:18:46.590572 18141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:46.590693 18141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6251ad272a3f450bb901d79d24701356, State: Initialized, Role: FOLLOWER
I20260926 02:18:46.591378 18141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6251ad272a3f450bb901d79d24701356" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35793 } }
I20260926 02:18:46.591611 18141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:46.591750 18141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:46.591900 18141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:46.594235 18141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6251ad272a3f450bb901d79d24701356" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35793 } }
I20260926 02:18:46.594836 18141 leader_election.cc:302] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6251ad272a3f450bb901d79d24701356; no voters:
I20260926 02:18:46.595496 18141 leader_election.cc:288] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:46.595619 18151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:46.595677 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:46.588917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:50733 (local address 127.0.0.1:35793)
0926 02:18:46.589117 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:46.589123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:46.589483 (+ 360us) server_negotiation.cc:407] Connection header received
0926 02:18:46.589678 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:46.589682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:46.589786 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:46.589871 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:46.590437 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:46.591373 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:46.592875 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:46.593423 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:46.593543 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:46.594832 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:46.594837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:46.594841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:46.594861 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:46.594922 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:46.594939 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:46.595041 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:46.595237 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 02:18:46.595297 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:18:46.596282 18151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [term 1 LEADER]: Becoming Leader. State: Replica: 6251ad272a3f450bb901d79d24701356, State: Running, Role: LEADER
I20260926 02:18:46.597074 18151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6251ad272a3f450bb901d79d24701356" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35793 } }
I20260926 02:18:46.599334 18141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:46.600752 18152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6251ad272a3f450bb901d79d24701356. Latest consensus state: current_term: 1 leader_uuid: "6251ad272a3f450bb901d79d24701356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6251ad272a3f450bb901d79d24701356" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35793 } } }
I20260926 02:18:46.600710 18153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6251ad272a3f450bb901d79d24701356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6251ad272a3f450bb901d79d24701356" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35793 } } }
I20260926 02:18:46.601404 18153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:46.602958 18152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:46.604206 18156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:46.611304 18156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:46.621953 18156 catalog_manager.cc:1347] Generated new cluster ID: b16a6ac87ad94b1ca92efe812e37640b
I20260926 02:18:46.622081 18156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:46.634534 18156 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:46.636619 18156 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:46.661053 18156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6251ad272a3f450bb901d79d24701356: Generated new TSK 0
I20260926 02:18:46.662273 18156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:46.675977 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18074
W20260926 02:18:46.677289 18136 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:37185
W20260926 02:18:46.678056 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18074: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:18:46.678198 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 18074
W20260926 02:18:46.692133 18059 connection.cc:535] client connection to 127.0.0.1:35793 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35793): Connection reset by peer (error 104)
2026-09-26T02:18:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1838 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:46 dist-test-slave-hllj krb5kdc[18172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:46 dist-test-slave-hllj krb5kdc[18172](info): set up 2 sockets
Sep 26 02:18:46 dist-test-slave-hllj krb5kdc[18172](info): commencing operation
krb5kdc: starting...
W20260926 02:18:47.706625 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:47 dist-test-slave-hllj krb5kdc[18172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:47.965296 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:48.296965 18185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:48.297325 18185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:48.297453 18185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:48.304756 18185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:48.304910 18185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:48.304986 18185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:48.305063 18185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:48.305121 18185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:48.315263 18185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:48.316629 18185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:48.318097 18185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:48.331647 18190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:48.331681 18191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:48.331707 18193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:48.332964 18185 server_base.cc:1034] running on GCE node
I20260926 02:18:48.333817 18185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:48.335336 18185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:48.336560 18185 hybrid_clock.cc:648] HybridClock initialized: now 1790389128336507 us; error 39 us; skew 500 ppm
I20260926 02:18:48.336936 18185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:48 dist-test-slave-hllj krb5kdc[18172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:48.348268 18185 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:48.350126 18185 webserver.cc:460] Webserver started at http://127.0.0.1:45689/ using document root <none> and password file <none>
I20260926 02:18:48.350865 18185 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:48.350991 18185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:48.351310 18185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:48.354470 18185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "3c510ca9397e463ea8f133cbc918874c"
format_stamp: "Formatted at 2026-09-26 02:18:48 on dist-test-slave-hllj"
server_key: "609a8ecefa5eeef7289f10f7b76a308f"
server_key_iv: "8b83ebeb33ce1de00cd7ec53a9eb4214"
server_key_version: "encryptionkey@0"
I20260926 02:18:48.355260 18185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "3c510ca9397e463ea8f133cbc918874c"
format_stamp: "Formatted at 2026-09-26 02:18:48 on dist-test-slave-hllj"
server_key: "609a8ecefa5eeef7289f10f7b76a308f"
server_key_iv: "8b83ebeb33ce1de00cd7ec53a9eb4214"
server_key_version: "encryptionkey@0"
I20260926 02:18:48.360720 18185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:18:48.364508 18200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:48.366259 18185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:18:48.366464 18185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "3c510ca9397e463ea8f133cbc918874c"
format_stamp: "Formatted at 2026-09-26 02:18:48 on dist-test-slave-hllj"
server_key: "609a8ecefa5eeef7289f10f7b76a308f"
server_key_iv: "8b83ebeb33ce1de00cd7ec53a9eb4214"
server_key_version: "encryptionkey@0"
I20260926 02:18:48.366685 18185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:48.373293 18185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:48.378114 18185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:48.378419 18185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:48.396329 18185 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43911
I20260926 02:18:48.396389 18251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43911 every 8 connection(s)
I20260926 02:18:48.397814 18185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:48.400209 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 18185
I20260926 02:18:48.400411 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:48.400950 29282 external_mini_cluster.cc:1428] Setting key 4ab0a4e4d074c4dd02b53add9d401aa5
I20260926 02:18:48.402612 18252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:48.409783 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:48.403113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:48251 (local address 127.0.0.1:43911)
0926 02:18:48.403938 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:48.403951 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:48.403995 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:18:48.405671 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:48.405698 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:48.406145 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:48.406591 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:48.407807 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:48.407814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:48.407832 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:48.407850 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:48.407950 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:48.407974 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:48.408194 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:48.408316 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 02:18:48.408563 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":358,"threads_started":1}
I20260926 02:18:48.414103 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:48.416633 18252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c: Bootstrap starting.
I20260926 02:18:48.421164 18252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:48.422233 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:48.419608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:44055 (local address 127.0.0.1:43911)
0926 02:18:48.419813 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:48.419818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:48.420182 (+ 364us) server_negotiation.cc:407] Connection header received
0926 02:18:48.420391 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:48.420395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:48.420446 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:48.420537 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:48.421580 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:48.421585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:48.421589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:48.421609 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:48.421657 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:48.421674 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:48.421780 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:48.421896 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 02:18:48.421948 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 02:18:48.422715 18252 log.cc:824] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:48.426083 18252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c: No bootstrap required, opened a new log
I20260926 02:18:48.434321 18252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c510ca9397e463ea8f133cbc918874c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260926 02:18:48.434713 18252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:48.434824 18252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c510ca9397e463ea8f133cbc918874c, State: Initialized, Role: FOLLOWER
I20260926 02:18:48.435416 18252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c510ca9397e463ea8f133cbc918874c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260926 02:18:48.435645 18252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:48.435777 18252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:48.435945 18252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:48.438378 18252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c510ca9397e463ea8f133cbc918874c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260926 02:18:48.438946 18252 leader_election.cc:302] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c510ca9397e463ea8f133cbc918874c; no voters:
I20260926 02:18:48.439524 18252 leader_election.cc:288] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:48.439663 18262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:48.440274 18262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [term 1 LEADER]: Becoming Leader. State: Replica: 3c510ca9397e463ea8f133cbc918874c, State: Running, Role: LEADER
I20260926 02:18:48.440819 18262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c510ca9397e463ea8f133cbc918874c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260926 02:18:48.441562 18252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:48.443933 18263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c510ca9397e463ea8f133cbc918874c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c510ca9397e463ea8f133cbc918874c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } } }
I20260926 02:18:48.444437 18263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:48.444037 18264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c510ca9397e463ea8f133cbc918874c. Latest consensus state: current_term: 1 leader_uuid: "3c510ca9397e463ea8f133cbc918874c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c510ca9397e463ea8f133cbc918874c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } } }
I20260926 02:18:48.444806 18264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:48.449798 18271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:48.455008 18271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:48.464677 18271 catalog_manager.cc:1347] Generated new cluster ID: ae091c91fb2542679f05999bdca0a3e0
I20260926 02:18:48.464838 18271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:48.475864 18271 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:48.476454 18271 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:48.487385 18271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c510ca9397e463ea8f133cbc918874c: Generated new TSK 0
I20260926 02:18:48.488319 18271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:48.504518 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18185
W20260926 02:18:48.506433 18247 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:48251
W20260926 02:18:48.507236 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18185: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:18:48.507422 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 18185
2026-09-26T02:18:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1829 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:48 dist-test-slave-hllj krb5kdc[18283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:48 dist-test-slave-hllj krb5kdc[18283](info): set up 2 sockets
Sep 26 02:18:48 dist-test-slave-hllj krb5kdc[18283](info): commencing operation
krb5kdc: starting...
W20260926 02:18:49.538717 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:49 dist-test-slave-hllj krb5kdc[18283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:49.796826 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:50.132840 18296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:50.133200 18296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:50.133344 18296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:50.140966 18296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:50.141156 18296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:50.141265 18296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:50.141350 18296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:50.141410 18296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:50.151626 18296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:50.152998 18296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:50.154451 18296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:50.166143 18301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:50.166182 18304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:50.166206 18302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:50.167060 18296 server_base.cc:1034] running on GCE node
I20260926 02:18:50.167872 18296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:50.169484 18296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:50.170708 18296 hybrid_clock.cc:648] HybridClock initialized: now 1790389130170660 us; error 47 us; skew 500 ppm
I20260926 02:18:50.171126 18296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:50 dist-test-slave-hllj krb5kdc[18283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:50.174324 18296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:50.176013 18296 webserver.cc:460] Webserver started at http://127.0.0.1:33435/ using document root <none> and password file <none>
I20260926 02:18:50.176726 18296 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:50.176862 18296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:50.177202 18296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:50.180377 18296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "5f622195fb3c413fb671daea48bc18ef"
format_stamp: "Formatted at 2026-09-26 02:18:50 on dist-test-slave-hllj"
server_key: "db6109c39b9155e27a834f10444f12a1"
server_key_iv: "032559c5f58568e397bb865a2606b440"
server_key_version: "encryptionkey@0"
I20260926 02:18:50.181268 18296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "5f622195fb3c413fb671daea48bc18ef"
format_stamp: "Formatted at 2026-09-26 02:18:50 on dist-test-slave-hllj"
server_key: "db6109c39b9155e27a834f10444f12a1"
server_key_iv: "032559c5f58568e397bb865a2606b440"
server_key_version: "encryptionkey@0"
I20260926 02:18:50.187005 18296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:18:50.190812 18311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:50.192380 18296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:18:50.192560 18296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "5f622195fb3c413fb671daea48bc18ef"
format_stamp: "Formatted at 2026-09-26 02:18:50 on dist-test-slave-hllj"
server_key: "db6109c39b9155e27a834f10444f12a1"
server_key_iv: "032559c5f58568e397bb865a2606b440"
server_key_version: "encryptionkey@0"
I20260926 02:18:50.192766 18296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:50.212164 18296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:50.216784 18296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:50.217070 18296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:50.235484 18296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37863
I20260926 02:18:50.235512 18362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37863 every 8 connection(s)
I20260926 02:18:50.236936 18296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:50.241060 18363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:50.242918 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 18296
I20260926 02:18:50.243081 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:50.243539 29282 external_mini_cluster.cc:1428] Setting key f14b23e9b1bb7fc850a9653a6e65388b
I20260926 02:18:50.255523 18363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef: Bootstrap starting.
Sep 26 02:18:50 dist-test-slave-hllj krb5kdc[18283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:18:50.259575 18363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:50.260807 18363 log.cc:824] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:50.264118 18363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef: No bootstrap required, opened a new log
I20260926 02:18:50.268769 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:50.245432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:38711 (local address 127.0.0.1:37863)
0926 02:18:50.246236 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:50.246249 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:50.246295 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:18:50.247958 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:50.247984 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:50.248443 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:50.248796 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:50.250847 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:50.252248 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:50.254227 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:50.254991 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:50.255138 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:50.260266 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:50.260276 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:50.260293 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:50.260343 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:50.263808 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:50.264961 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:50.264970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:50.264977 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:50.265197 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:50.265725 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:50.265729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:50.265731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:50.266324 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:50.266591 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:50.267235 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 02:18:50.267623 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":348,"threads_started":1}
I20260926 02:18:50.273496 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:50.273490 18363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f622195fb3c413fb671daea48bc18ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260926 02:18:50.273991 18363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:50.274092 18363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f622195fb3c413fb671daea48bc18ef, State: Initialized, Role: FOLLOWER
I20260926 02:18:50.274773 18363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f622195fb3c413fb671daea48bc18ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260926 02:18:50.275071 18363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:50.275197 18363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:50.275352 18363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:50.278623 18363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f622195fb3c413fb671daea48bc18ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260926 02:18:50.279284 18363 leader_election.cc:302] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f622195fb3c413fb671daea48bc18ef; no voters:
I20260926 02:18:50.280006 18363 leader_election.cc:288] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:50.280100 18373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:50.282423 18363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:50.286796 18373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [term 1 LEADER]: Becoming Leader. State: Replica: 5f622195fb3c413fb671daea48bc18ef, State: Running, Role: LEADER
I20260926 02:18:50.287519 18373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f622195fb3c413fb671daea48bc18ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } }
I20260926 02:18:50.290516 18374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f622195fb3c413fb671daea48bc18ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f622195fb3c413fb671daea48bc18ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } } }
I20260926 02:18:50.290933 18374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:50.291477 18375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f622195fb3c413fb671daea48bc18ef. Latest consensus state: current_term: 1 leader_uuid: "5f622195fb3c413fb671daea48bc18ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f622195fb3c413fb671daea48bc18ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37863 } } }
I20260926 02:18:50.291715 18375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:50.299364 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:50.278910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:34495 (local address 127.0.0.1:37863)
0926 02:18:50.279112 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:50.279118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:50.279471 (+ 353us) server_negotiation.cc:407] Connection header received
0926 02:18:50.279642 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:50.279646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:50.279716 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:50.279843 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:50.281885 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:50.283212 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:50.285176 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:50.287010 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:50.287149 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:50.290723 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:50.290733 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:50.290738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:50.290800 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:50.294706 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:50.295948 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:50.295955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:50.295959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:50.296150 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:50.296753 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:50.296759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:50.296777 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:50.297213 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:50.298545 (+ 1332us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:50.298743 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 02:18:50.298962 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:18:50.301625 18386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:50.306020 18386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:50.315625 18386 catalog_manager.cc:1347] Generated new cluster ID: e7a319df10e4492b82a23dc53ca991d8
I20260926 02:18:50.315752 18386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:50.332093 18386 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:50.334137 18386 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:50.344957 18386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f622195fb3c413fb671daea48bc18ef: Generated new TSK 0
I20260926 02:18:50.345956 18386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:50.354142 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18296
I20260926 02:18:50.355347 18358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:50.444120 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 18296
W20260926 02:18:50.457829 18281 connection.cc:535] client connection to 127.0.0.1:37863 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37863): Connection reset by peer (error 104)
2026-09-26T02:18:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:50 dist-test-slave-hllj krb5kdc[18395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:50 dist-test-slave-hllj krb5kdc[18395](info): set up 2 sockets
Sep 26 02:18:50 dist-test-slave-hllj krb5kdc[18395](info): commencing operation
krb5kdc: starting...
W20260926 02:18:51.474705 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:51 dist-test-slave-hllj krb5kdc[18395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:51.733870 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:52.071573 18408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:52.071905 18408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:52.072001 18408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:52.079680 18408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:52.079824 18408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:52.079881 18408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:52.079946 18408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:52.079989 18408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:52.090406 18408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40295
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:52.091744 18408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:52.093046 18408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:52.105417 18413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:52.105356 18416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:52.105400 18414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:52.106202 18408 server_base.cc:1034] running on GCE node
I20260926 02:18:52.107067 18408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:52.108667 18408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:52.109879 18408 hybrid_clock.cc:648] HybridClock initialized: now 1790389132109826 us; error 55 us; skew 500 ppm
I20260926 02:18:52.110272 18408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:52 dist-test-slave-hllj krb5kdc[18395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:52.113507 18408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:52.115149 18408 webserver.cc:460] Webserver started at http://127.0.0.1:33333/ using document root <none> and password file <none>
I20260926 02:18:52.115876 18408 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:52.116000 18408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:52.116318 18408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:52.119454 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "0689f1870d944baa86bc65082ddde50b"
format_stamp: "Formatted at 2026-09-26 02:18:52 on dist-test-slave-hllj"
server_key: "1b282fa81b23c41b2bf0f024b5d5d3a6"
server_key_iv: "2cbe2418aa829f665d81b976b0a4fdc7"
server_key_version: "encryptionkey@0"
I20260926 02:18:52.120240 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "0689f1870d944baa86bc65082ddde50b"
format_stamp: "Formatted at 2026-09-26 02:18:52 on dist-test-slave-hllj"
server_key: "1b282fa81b23c41b2bf0f024b5d5d3a6"
server_key_iv: "2cbe2418aa829f665d81b976b0a4fdc7"
server_key_version: "encryptionkey@0"
I20260926 02:18:52.125840 18408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:18:52.129736 18423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:52.131590 18408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:18:52.131863 18408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "0689f1870d944baa86bc65082ddde50b"
format_stamp: "Formatted at 2026-09-26 02:18:52 on dist-test-slave-hllj"
server_key: "1b282fa81b23c41b2bf0f024b5d5d3a6"
server_key_iv: "2cbe2418aa829f665d81b976b0a4fdc7"
server_key_version: "encryptionkey@0"
I20260926 02:18:52.132082 18408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:52.146903 18408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:52.151734 18408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:52.152030 18408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:52.170323 18408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40295
I20260926 02:18:52.170346 18474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40295 every 8 connection(s)
I20260926 02:18:52.171782 18408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:52.175771 18475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:52.181771 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 18408
I20260926 02:18:52.181960 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:52.182431 29282 external_mini_cluster.cc:1428] Setting key 310205823109ee3101dada0e9ffff98c
I20260926 02:18:52.187589 18475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b: Bootstrap starting.
I20260926 02:18:52.191536 18475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:52.192817 18475 log.cc:824] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:52.196166 18475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b: No bootstrap required, opened a new log
I20260926 02:18:52.196978 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:52.184378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:57239 (local address 127.0.0.1:40295)
0926 02:18:52.185297 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:52.185310 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:52.185364 (+ 54us) server_negotiation.cc:407] Connection header received
0926 02:18:52.187181 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:52.187212 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:52.187702 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:52.188115 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:52.188769 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:52.190240 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:52.192189 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:52.193006 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:52.193211 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:52.194532 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:52.194541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:52.194559 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:52.195134 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:52.195213 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:52.195246 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:52.195378 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:52.195637 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 02:18:52.195898 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":400,"threads_started":1}
I20260926 02:18:52.200974 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:52.205665 18475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0689f1870d944baa86bc65082ddde50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260926 02:18:52.206163 18475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:52.206318 18475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0689f1870d944baa86bc65082ddde50b, State: Initialized, Role: FOLLOWER
I20260926 02:18:52.207091 18475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0689f1870d944baa86bc65082ddde50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260926 02:18:52.207425 18475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:52.207576 18475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:52.207791 18475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:52.211043 18475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0689f1870d944baa86bc65082ddde50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260926 02:18:52.211704 18475 leader_election.cc:302] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0689f1870d944baa86bc65082ddde50b; no voters:
I20260926 02:18:52.212450 18475 leader_election.cc:288] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:52.212607 18485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:52.214277 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:52.206366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:60091 (local address 127.0.0.1:40295)
0926 02:18:52.206580 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:52.206588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:52.207065 (+ 477us) server_negotiation.cc:407] Connection header received
0926 02:18:52.207282 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:52.207286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:52.207383 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:52.207487 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:52.208100 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:52.208960 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:52.210914 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:52.211445 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:52.211547 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:52.212910 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:52.212916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:52.212921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:52.212949 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:52.213046 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:52.213077 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:52.213353 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:52.213812 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 02:18:52.213878 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260926 02:18:52.214687 18475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:52.214706 18485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [term 1 LEADER]: Becoming Leader. State: Replica: 0689f1870d944baa86bc65082ddde50b, State: Running, Role: LEADER
I20260926 02:18:52.215418 18485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0689f1870d944baa86bc65082ddde50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } }
I20260926 02:18:52.218307 18486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0689f1870d944baa86bc65082ddde50b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0689f1870d944baa86bc65082ddde50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } } }
I20260926 02:18:52.218155 18487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0689f1870d944baa86bc65082ddde50b. Latest consensus state: current_term: 1 leader_uuid: "0689f1870d944baa86bc65082ddde50b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0689f1870d944baa86bc65082ddde50b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40295 } } }
I20260926 02:18:52.218641 18487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:52.219193 18486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:52.219513 18495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:52.224597 18495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:52.233585 18495 catalog_manager.cc:1347] Generated new cluster ID: dbba6a76d5404d598a9970d6aa5b7c5a
I20260926 02:18:52.233727 18495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:52.246187 18495 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:52.247629 18495 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:52.259519 18495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0689f1870d944baa86bc65082ddde50b: Generated new TSK 0
I20260926 02:18:52.260452 18495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:52.268226 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18408
W20260926 02:18:52.269359 18470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:57239
W20260926 02:18:52.270097 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18408: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:18:52.270248 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 18408
W20260926 02:18:52.283763 18393 connection.cc:535] client connection to 127.0.0.1:40295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40295): Connection reset by peer (error 104)
2026-09-26T02:18:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1826 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:52 dist-test-slave-hllj krb5kdc[18506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:52 dist-test-slave-hllj krb5kdc[18506](info): set up 2 sockets
Sep 26 02:18:52 dist-test-slave-hllj krb5kdc[18506](info): commencing operation
krb5kdc: starting...
W20260926 02:18:53.302735 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:53 dist-test-slave-hllj krb5kdc[18506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:18:53.560721 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:53.898658 18519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:53.899003 18519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:53.899147 18519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:53.906723 18519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:53.906863 18519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:53.906948 18519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:53.907030 18519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:53.907086 18519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:53.917469 18519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:53.918838 18519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:53.920246 18519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:53.932598 18525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:53.932689 18527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:53.932598 18524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:53.933475 18519 server_base.cc:1034] running on GCE node
I20260926 02:18:53.934288 18519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:53.935868 18519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:53.937098 18519 hybrid_clock.cc:648] HybridClock initialized: now 1790389133937044 us; error 48 us; skew 500 ppm
I20260926 02:18:53.937511 18519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:53 dist-test-slave-hllj krb5kdc[18506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:53.944756 18519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:18:53.946552 18519 webserver.cc:460] Webserver started at http://127.0.0.1:37697/ using document root <none> and password file <none>
I20260926 02:18:53.947319 18519 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:53.947453 18519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:53.947769 18519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:53.951035 18519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "ab681b97ef184dd0833065dfe4752589"
format_stamp: "Formatted at 2026-09-26 02:18:53 on dist-test-slave-hllj"
server_key: "f5fc1a81cb931f0a91bc5912d0ae2020"
server_key_iv: "607f722f1feb3868e5ac432e9e4ac100"
server_key_version: "encryptionkey@0"
I20260926 02:18:53.951840 18519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "ab681b97ef184dd0833065dfe4752589"
format_stamp: "Formatted at 2026-09-26 02:18:53 on dist-test-slave-hllj"
server_key: "f5fc1a81cb931f0a91bc5912d0ae2020"
server_key_iv: "607f722f1feb3868e5ac432e9e4ac100"
server_key_version: "encryptionkey@0"
I20260926 02:18:53.957602 18519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:18:53.961807 18534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:53.963482 18519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 02:18:53.963683 18519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "ab681b97ef184dd0833065dfe4752589"
format_stamp: "Formatted at 2026-09-26 02:18:53 on dist-test-slave-hllj"
server_key: "f5fc1a81cb931f0a91bc5912d0ae2020"
server_key_iv: "607f722f1feb3868e5ac432e9e4ac100"
server_key_version: "encryptionkey@0"
I20260926 02:18:53.963912 18519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:53.970151 18519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:53.974442 18519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:53.974740 18519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:53.993676 18519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37071
I20260926 02:18:53.993781 18585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37071 every 8 connection(s)
I20260926 02:18:53.995157 18519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:53.996263 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 18519
I20260926 02:18:53.996431 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:53.996861 29282 external_mini_cluster.cc:1428] Setting key dfd630abe1b93520bb967338fa840a0a
I20260926 02:18:53.999811 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:54.004942 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:53.998791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:53253 (local address 127.0.0.1:37071)
0926 02:18:53.999601 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:53.999615 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:53.999669 (+ 54us) server_negotiation.cc:407] Connection header received
0926 02:18:54.001438 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:54.001463 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:54.001830 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:54.002264 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:54.003277 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:54.003283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:54.003298 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:54.003315 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:54.003387 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:54.003410 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:54.003628 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:54.003764 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 02:18:54.003966 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":343,"threads_started":1}
I20260926 02:18:54.009222 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:54.012060 18586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589: Bootstrap starting.
I20260926 02:18:54.016018 18586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:54.016331 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:54.013740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:41007 (local address 127.0.0.1:37071)
0926 02:18:54.013937 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:54.013949 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:54.014340 (+ 391us) server_negotiation.cc:407] Connection header received
0926 02:18:54.014536 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:54.014540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:54.014592 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:54.014729 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:54.015663 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:54.015668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:54.015671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:54.015687 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:54.015740 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:54.015756 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:54.015850 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:54.016005 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 02:18:54.016058 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:18:54.017467 18586 log.cc:824] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:54.020339 18586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589: No bootstrap required, opened a new log
I20260926 02:18:54.027999 18586 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681b97ef184dd0833065dfe4752589" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260926 02:18:54.028394 18586 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:54.028470 18586 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab681b97ef184dd0833065dfe4752589, State: Initialized, Role: FOLLOWER
I20260926 02:18:54.028990 18586 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681b97ef184dd0833065dfe4752589" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260926 02:18:54.029229 18586 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:54.029341 18586 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:54.029510 18586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:54.031665 18586 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681b97ef184dd0833065dfe4752589" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260926 02:18:54.032223 18586 leader_election.cc:302] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab681b97ef184dd0833065dfe4752589; no voters:
I20260926 02:18:54.032909 18586 leader_election.cc:288] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:54.033049 18596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:54.033717 18596 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [term 1 LEADER]: Becoming Leader. State: Replica: ab681b97ef184dd0833065dfe4752589, State: Running, Role: LEADER
I20260926 02:18:54.034296 18596 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681b97ef184dd0833065dfe4752589" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260926 02:18:54.035086 18586 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:54.037174 18598 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab681b97ef184dd0833065dfe4752589. Latest consensus state: current_term: 1 leader_uuid: "ab681b97ef184dd0833065dfe4752589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681b97ef184dd0833065dfe4752589" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } } }
I20260926 02:18:54.037379 18597 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab681b97ef184dd0833065dfe4752589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681b97ef184dd0833065dfe4752589" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } } }
I20260926 02:18:54.037604 18598 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:54.037618 18597 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:54.038492 18604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:54.044827 18604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:54.055193 18604 catalog_manager.cc:1347] Generated new cluster ID: 3f773d94170640cda33f1c0558e6c489
I20260926 02:18:54.055318 18604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:54.084213 18604 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:54.084859 18604 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:54.097232 18604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab681b97ef184dd0833065dfe4752589: Generated new TSK 0
I20260926 02:18:54.098423 18604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:54.111840 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18519
W20260926 02:18:54.113190 18581 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:53253
W20260926 02:18:54.113901 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18519: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:18:54.114023 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 18519
2026-09-26T02:18:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1847 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:54 dist-test-slave-hllj krb5kdc[18617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:54 dist-test-slave-hllj krb5kdc[18617](info): set up 2 sockets
Sep 26 02:18:54 dist-test-slave-hllj krb5kdc[18617](info): commencing operation
krb5kdc: starting...
W20260926 02:18:55.154744 29282 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:55 dist-test-slave-hllj krb5kdc[18617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:18:55.412115 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:40465
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:40465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:55.752476 18630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:55.752851 18630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:55.752982 18630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:55.760763 18630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:55.760905 18630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:55.760972 18630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:55.761063 18630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:55.761118 18630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:55.771710 18630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:40465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:40465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:55.773108 18630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:55.774492 18630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:55.786906 18635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:55.786904 18638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:55.786895 18636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:55.787685 18630 server_base.cc:1034] running on GCE node
I20260926 02:18:55.788580 18630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:55.790270 18630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:55.791491 18630 hybrid_clock.cc:648] HybridClock initialized: now 1790389135791438 us; error 48 us; skew 500 ppm
I20260926 02:18:55.791980 18630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:55 dist-test-slave-hllj krb5kdc[18617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389135, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:55.795470 18630 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:18:55.797317 18630 webserver.cc:460] Webserver started at http://127.28.152.190:33177/ using document root <none> and password file <none>
I20260926 02:18:55.798048 18630 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:55.798183 18630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:55.798492 18630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:55.801743 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "493081a84b29480ba3dfe8fdbc46f4f3"
format_stamp: "Formatted at 2026-09-26 02:18:55 on dist-test-slave-hllj"
server_key: "4d9d1e92e993351dc79102fa0d29bfbe"
server_key_iv: "033d4e34947a81aa433c80f24a811658"
server_key_version: "encryptionkey@0"
I20260926 02:18:55.802532 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "493081a84b29480ba3dfe8fdbc46f4f3"
format_stamp: "Formatted at 2026-09-26 02:18:55 on dist-test-slave-hllj"
server_key: "4d9d1e92e993351dc79102fa0d29bfbe"
server_key_iv: "033d4e34947a81aa433c80f24a811658"
server_key_version: "encryptionkey@0"
I20260926 02:18:55.808434 18630 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:18:55.812445 18645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:55.814163 18630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:18:55.814376 18630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "493081a84b29480ba3dfe8fdbc46f4f3"
format_stamp: "Formatted at 2026-09-26 02:18:55 on dist-test-slave-hllj"
server_key: "4d9d1e92e993351dc79102fa0d29bfbe"
server_key_iv: "033d4e34947a81aa433c80f24a811658"
server_key_version: "encryptionkey@0"
I20260926 02:18:55.814580 18630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:55.836311 18630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:55.841010 18630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:55.841342 18630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:55.862529 18630 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:40465
I20260926 02:18:55.862536 18696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:40465 every 8 connection(s)
I20260926 02:18:55.863930 18630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:55.867498 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 18630
I20260926 02:18:55.867643 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:55.867841 18697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:55.868094 29282 external_mini_cluster.cc:1428] Setting key 67b734b8c3b91f37edbb28d027039594
I20260926 02:18:55.882727 18697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3: Bootstrap starting.
Sep 26 02:18:55 dist-test-slave-hllj krb5kdc[18617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:18:55.886870 18697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:55.888422 18697 log.cc:824] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:55.892688 18697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3: No bootstrap required, opened a new log
I20260926 02:18:55.894174 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:55.869961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46649 (local address 127.28.152.190:40465)
0926 02:18:55.870787 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:55.870808 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:55.870858 (+ 50us) server_negotiation.cc:407] Connection header received
0926 02:18:55.872787 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:55.872820 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:55.873492 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:55.874007 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:55.875387 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:55.877385 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:55.878730 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:55.879905 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:55.880128 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:55.884556 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:55.884571 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:55.884601 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:55.884661 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:55.888856 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:55.890291 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:55.890304 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:55.890314 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:55.890577 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:55.891063 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:55.891068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:55.891072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:55.891846 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:55.892179 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:55.892592 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 02:18:55.892977 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":328,"threads_started":1}
I20260926 02:18:55.898507 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:55.902704 18697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493081a84b29480ba3dfe8fdbc46f4f3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40465 } }
I20260926 02:18:55.903277 18697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:55.903398 18697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493081a84b29480ba3dfe8fdbc46f4f3, State: Initialized, Role: FOLLOWER
I20260926 02:18:55.904158 18697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493081a84b29480ba3dfe8fdbc46f4f3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40465 } }
I20260926 02:18:55.904724 18697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:55.904913 18697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:55.905200 18697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:55.908294 18697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493081a84b29480ba3dfe8fdbc46f4f3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40465 } }
I20260926 02:18:55.909065 18697 leader_election.cc:302] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 493081a84b29480ba3dfe8fdbc46f4f3; no voters:
I20260926 02:18:55.909883 18697 leader_election.cc:288] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:55.909991 18707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:55.911198 18707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [term 1 LEADER]: Becoming Leader. State: Replica: 493081a84b29480ba3dfe8fdbc46f4f3, State: Running, Role: LEADER
I20260926 02:18:55.911906 18707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493081a84b29480ba3dfe8fdbc46f4f3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40465 } }
I20260926 02:18:55.912264 18697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:55.915118 18709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 493081a84b29480ba3dfe8fdbc46f4f3. Latest consensus state: current_term: 1 leader_uuid: "493081a84b29480ba3dfe8fdbc46f4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493081a84b29480ba3dfe8fdbc46f4f3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40465 } } }
I20260926 02:18:55.915544 18709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:55.916235 18708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "493081a84b29480ba3dfe8fdbc46f4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493081a84b29480ba3dfe8fdbc46f4f3" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 40465 } } }
I20260926 02:18:55.916563 18708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:55.918285 18714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:55.923698 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:55.904247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53945 (local address 127.28.152.190:40465)
0926 02:18:55.904496 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:55.904503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:55.904885 (+ 382us) server_negotiation.cc:407] Connection header received
0926 02:18:55.905108 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:55.905114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:55.905251 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:55.905358 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:55.907318 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:55.908203 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:55.910205 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:55.910973 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:55.911126 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:55.914847 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:55.914856 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:55.914860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:18:55.914913 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:55.918813 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:55.919921 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:55.919927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:55.919931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:55.920064 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:18:55.920603 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:18:55.920609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:18:55.920613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:18:55.921211 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:55.921568 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:55.923057 (+ 1489us) server_negotiation.cc:299] Negotiation successful
0926 02:18:55.923288 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:18:55.925669 18714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:55.936229 18714 catalog_manager.cc:1347] Generated new cluster ID: 0fa7f56e475e4cd798414231e195efd2
I20260926 02:18:55.936365 18714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:55.961781 18714 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:55.963343 18714 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:55.971372 18714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 493081a84b29480ba3dfe8fdbc46f4f3: Generated new TSK 0
I20260926 02:18:55.972311 18714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:55.984381 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18630
I20260926 02:18:55.985680 18692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:18:56.078044 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 18630
W20260926 02:18:56.092208 18615 connection.cc:535] client connection to 127.28.152.190:40465 recv error: Network error: failed to read from TLS socket (remote: 127.28.152.190:40465): Connection reset by peer (error 104)
2026-09-26T02:18:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:56 dist-test-slave-hllj krb5kdc[18729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:56 dist-test-slave-hllj krb5kdc[18729](info): set up 2 sockets
Sep 26 02:18:56 dist-test-slave-hllj krb5kdc[18729](info): commencing operation
krb5kdc: starting...
W20260926 02:18:57.102751 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:57 dist-test-slave-hllj krb5kdc[18729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:18:57.356671 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34153
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:34153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:57.690985 18742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:57.691324 18742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:57.691468 18742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:57.698967 18742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:57.699115 18742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:57.699195 18742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:57.699277 18742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:57.699337 18742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:57.709599 18742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:34153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:34153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:57.710956 18742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:57.712263 18742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:57.724278 18747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:57.724332 18748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:57.724341 18750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:57.725072 18742 server_base.cc:1034] running on GCE node
I20260926 02:18:57.725912 18742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:57.727434 18742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:57.728662 18742 hybrid_clock.cc:648] HybridClock initialized: now 1790389137728606 us; error 54 us; skew 500 ppm
I20260926 02:18:57.729033 18742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:57 dist-test-slave-hllj krb5kdc[18729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:57.732304 18742 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:18:57.733999 18742 webserver.cc:460] Webserver started at http://127.28.152.190:40473/ using document root <none> and password file <none>
I20260926 02:18:57.734746 18742 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:57.734874 18742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:57.735188 18742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:57.738454 18742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "4996e67cd19d44adabc904b86197bf46"
format_stamp: "Formatted at 2026-09-26 02:18:57 on dist-test-slave-hllj"
server_key: "ccfba2e4ca72773a7abcc5fddc462005"
server_key_iv: "da76105f9c7db76ac1d0129fe3dcfa68"
server_key_version: "encryptionkey@0"
I20260926 02:18:57.739295 18742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "4996e67cd19d44adabc904b86197bf46"
format_stamp: "Formatted at 2026-09-26 02:18:57 on dist-test-slave-hllj"
server_key: "ccfba2e4ca72773a7abcc5fddc462005"
server_key_iv: "da76105f9c7db76ac1d0129fe3dcfa68"
server_key_version: "encryptionkey@0"
I20260926 02:18:57.745306 18742 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:18:57.749166 18757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:57.750797 18742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:18:57.750977 18742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "4996e67cd19d44adabc904b86197bf46"
format_stamp: "Formatted at 2026-09-26 02:18:57 on dist-test-slave-hllj"
server_key: "ccfba2e4ca72773a7abcc5fddc462005"
server_key_iv: "da76105f9c7db76ac1d0129fe3dcfa68"
server_key_version: "encryptionkey@0"
I20260926 02:18:57.751190 18742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:57.772842 18742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:57.777380 18742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:57.777659 18742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:57.795598 18742 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:34153
I20260926 02:18:57.795646 18808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:34153 every 8 connection(s)
I20260926 02:18:57.796985 18742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:57.800913 18809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:57.801556 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 18742
I20260926 02:18:57.801749 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:57.802193 29282 external_mini_cluster.cc:1428] Setting key e6d188cee0585d105096efd7f66c0a2f
I20260926 02:18:57.814721 18809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46: Bootstrap starting.
I20260926 02:18:57.816336 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:57.804106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38687 (local address 127.28.152.190:34153)
0926 02:18:57.805015 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:57.805031 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:57.805084 (+ 53us) server_negotiation.cc:407] Connection header received
0926 02:18:57.806899 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:57.806929 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:57.807407 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:57.807793 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:57.808426 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:57.809809 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:57.811717 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:57.812471 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:57.812624 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:57.813909 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:57.813917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:57.813932 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:57.814511 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:57.814593 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:57.814620 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:57.814750 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:57.815086 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 02:18:57.815306 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":353,"threads_started":1}
I20260926 02:18:57.818120 18809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:57.819286 18809 log.cc:824] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:57.820261 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:57.822301 18809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46: No bootstrap required, opened a new log
I20260926 02:18:57.830596 18809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4996e67cd19d44adabc904b86197bf46" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34153 } }
I20260926 02:18:57.831166 18809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:57.831203 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:57.824764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47513 (local address 127.28.152.190:34153)
0926 02:18:57.824955 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:57.824963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:57.825349 (+ 386us) server_negotiation.cc:407] Connection header received
0926 02:18:57.825536 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:57.825540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:57.825624 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:57.825722 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:57.826270 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:57.827062 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:57.828453 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:18:57.828979 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:18:57.829123 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:18:57.830424 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:57.830429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:57.830433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:57.830453 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:57.830516 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:57.830536 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:57.830648 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:57.830841 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 02:18:57.830903 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:18:57.831293 18809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4996e67cd19d44adabc904b86197bf46, State: Initialized, Role: FOLLOWER
I20260926 02:18:57.831980 18809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4996e67cd19d44adabc904b86197bf46" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34153 } }
I20260926 02:18:57.832240 18809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:57.832372 18809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:57.832556 18809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:57.834959 18809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4996e67cd19d44adabc904b86197bf46" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34153 } }
I20260926 02:18:57.835494 18809 leader_election.cc:302] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4996e67cd19d44adabc904b86197bf46; no voters:
I20260926 02:18:57.836117 18809 leader_election.cc:288] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:57.836259 18819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:57.836899 18819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [term 1 LEADER]: Becoming Leader. State: Replica: 4996e67cd19d44adabc904b86197bf46, State: Running, Role: LEADER
I20260926 02:18:57.837615 18819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4996e67cd19d44adabc904b86197bf46" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34153 } }
I20260926 02:18:57.838280 18809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:57.840271 18821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4996e67cd19d44adabc904b86197bf46. Latest consensus state: current_term: 1 leader_uuid: "4996e67cd19d44adabc904b86197bf46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4996e67cd19d44adabc904b86197bf46" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34153 } } }
I20260926 02:18:57.840675 18821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:57.840677 18820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4996e67cd19d44adabc904b86197bf46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4996e67cd19d44adabc904b86197bf46" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 34153 } } }
I20260926 02:18:57.841007 18820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:57.842650 18826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:57.848289 18826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:57.859377 18826 catalog_manager.cc:1347] Generated new cluster ID: 9217fc8aa2c64138a702be680420ac4a
I20260926 02:18:57.859508 18826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:57.877749 18826 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:57.879065 18826 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:57.887827 18826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4996e67cd19d44adabc904b86197bf46: Generated new TSK 0
I20260926 02:18:57.888697 18826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:57.901937 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18742
W20260926 02:18:57.903208 18804 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38687
W20260926 02:18:57.904071 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18742: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:18:57.904225 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 18742
W20260926 02:18:57.917775 18727 connection.cc:535] client connection to 127.28.152.190:34153 recv error: Network error: failed to read from TLS socket (remote: 127.28.152.190:34153): Connection reset by peer (error 104)
2026-09-26T02:18:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1825 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:58 dist-test-slave-hllj krb5kdc[18840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:58 dist-test-slave-hllj krb5kdc[18840](info): set up 2 sockets
Sep 26 02:18:58 dist-test-slave-hllj krb5kdc[18840](info): commencing operation
krb5kdc: starting...
W20260926 02:18:58.926728 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:18:59 dist-test-slave-hllj krb5kdc[18840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:18:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:18:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:18:59.179925 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:38641
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:38641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:18:59.513851 18853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:18:59.514173 18853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:18:59.514294 18853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:18:59.521706 18853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:18:59.521833 18853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:18:59.521886 18853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:18:59.521968 18853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:18:59.522020 18853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:18:59.532423 18853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:38641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:38641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:18:59.534015 18853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:18:59.535391 18853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:18:59.547202 18858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:59.547289 18859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:18:59.547354 18861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:18:59.548480 18853 server_base.cc:1034] running on GCE node
I20260926 02:18:59.549345 18853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:18:59.550865 18853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:18:59.552098 18853 hybrid_clock.cc:648] HybridClock initialized: now 1790389139552054 us; error 56 us; skew 500 ppm
I20260926 02:18:59.552454 18853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:18:59 dist-test-slave-hllj krb5kdc[18840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:18:59.562637 18853 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:18:59.564316 18853 webserver.cc:460] Webserver started at http://127.28.152.190:42123/ using document root <none> and password file <none>
I20260926 02:18:59.565059 18853 fs_manager.cc:359] Metadata directory not provided
I20260926 02:18:59.565243 18853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:18:59.565562 18853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:18:59.568722 18853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "47767607b7804aa5934ffb12f7db56c7"
format_stamp: "Formatted at 2026-09-26 02:18:59 on dist-test-slave-hllj"
server_key: "26bb1fbc4c33b0261f5f4ae70daf13c9"
server_key_iv: "f60337ea06d7a7ead0350b057116d2c7"
server_key_version: "encryptionkey@0"
I20260926 02:18:59.569594 18853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "47767607b7804aa5934ffb12f7db56c7"
format_stamp: "Formatted at 2026-09-26 02:18:59 on dist-test-slave-hllj"
server_key: "26bb1fbc4c33b0261f5f4ae70daf13c9"
server_key_iv: "f60337ea06d7a7ead0350b057116d2c7"
server_key_version: "encryptionkey@0"
I20260926 02:18:59.575059 18853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:18:59.578931 18868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:18:59.580451 18853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 02:18:59.580626 18853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "47767607b7804aa5934ffb12f7db56c7"
format_stamp: "Formatted at 2026-09-26 02:18:59 on dist-test-slave-hllj"
server_key: "26bb1fbc4c33b0261f5f4ae70daf13c9"
server_key_iv: "f60337ea06d7a7ead0350b057116d2c7"
server_key_version: "encryptionkey@0"
I20260926 02:18:59.580835 18853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:18:59.586735 18853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:18:59.591161 18853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:18:59.591456 18853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:18:59.609515 18853 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:38641
I20260926 02:18:59.609540 18919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:38641 every 8 connection(s)
I20260926 02:18:59.611003 18853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:18:59.613999 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 18853
I20260926 02:18:59.614172 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:18:59.614595 29282 external_mini_cluster.cc:1428] Setting key 0c91359666199a0c357560cd278539e3
I20260926 02:18:59.615067 18920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:18:59.624382 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:59.616450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38475 (local address 127.28.152.190:38641)
0926 02:18:59.617318 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:59.617335 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:59.617387 (+ 52us) server_negotiation.cc:407] Connection header received
0926 02:18:59.620009 (+ 2622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:59.620044 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:59.620577 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:59.621107 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:59.622226 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:59.622236 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:59.622259 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:59.622277 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:59.622359 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:59.622387 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:59.622634 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:59.622779 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 02:18:59.623021 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":347,"threads_started":1}
I20260926 02:18:59.628551 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:18:59.630038 18920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7: Bootstrap starting.
I20260926 02:18:59.633477 18920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7: Neither blocks nor log segments found. Creating new log.
I20260926 02:18:59.634904 18920 log.cc:824] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7: Log is configured to *not* fsync() on all Append() calls
I20260926 02:18:59.635491 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:18:59.632859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43737 (local address 127.28.152.190:38641)
0926 02:18:59.633057 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:18:59.633064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:18:59.633592 (+ 528us) server_negotiation.cc:407] Connection header received
0926 02:18:59.633785 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:18:59.633788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:18:59.633843 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:18:59.633928 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:18:59.634857 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:18:59.634861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:18:59.634864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:18:59.634879 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:18:59.634922 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:18:59.634937 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:18:59.635035 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 02:18:59.635171 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 02:18:59.635221 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:18:59.638121 18920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7: No bootstrap required, opened a new log
I20260926 02:18:59.646121 18920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47767607b7804aa5934ffb12f7db56c7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38641 } }
I20260926 02:18:59.646557 18920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:18:59.646638 18920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47767607b7804aa5934ffb12f7db56c7, State: Initialized, Role: FOLLOWER
I20260926 02:18:59.647172 18920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47767607b7804aa5934ffb12f7db56c7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38641 } }
I20260926 02:18:59.647372 18920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:18:59.647487 18920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:18:59.647610 18920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:18:59.649906 18920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47767607b7804aa5934ffb12f7db56c7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38641 } }
I20260926 02:18:59.650431 18920 leader_election.cc:302] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47767607b7804aa5934ffb12f7db56c7; no voters:
I20260926 02:18:59.651031 18920 leader_election.cc:288] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:18:59.651183 18930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:18:59.652321 18930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [term 1 LEADER]: Becoming Leader. State: Replica: 47767607b7804aa5934ffb12f7db56c7, State: Running, Role: LEADER
I20260926 02:18:59.653083 18930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47767607b7804aa5934ffb12f7db56c7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38641 } }
I20260926 02:18:59.653342 18920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:18:59.656225 18932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47767607b7804aa5934ffb12f7db56c7. Latest consensus state: current_term: 1 leader_uuid: "47767607b7804aa5934ffb12f7db56c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47767607b7804aa5934ffb12f7db56c7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38641 } } }
I20260926 02:18:59.656199 18931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47767607b7804aa5934ffb12f7db56c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47767607b7804aa5934ffb12f7db56c7" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38641 } } }
I20260926 02:18:59.656702 18931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:59.656678 18932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:18:59.657590 18939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:18:59.663780 18939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:18:59.674588 18939 catalog_manager.cc:1347] Generated new cluster ID: 8ee74e37929b4d6e94083349eca36e53
I20260926 02:18:59.674721 18939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:18:59.691740 18939 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:18:59.692202 18939 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:18:59.699465 18939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47767607b7804aa5934ffb12f7db56c7: Generated new TSK 0
I20260926 02:18:59.700371 18939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:18:59.713493 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18853
W20260926 02:18:59.715301 18915 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38475
W20260926 02:18:59.716005 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18853: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:18:59.716157 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 18853
2026-09-26T02:18:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1813 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:18:59 dist-test-slave-hllj krb5kdc[18951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:18:59 dist-test-slave-hllj krb5kdc[18951](info): set up 2 sockets
Sep 26 02:18:59 dist-test-slave-hllj krb5kdc[18951](info): commencing operation
krb5kdc: starting...
W20260926 02:19:00.762802 29282 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:19:00 dist-test-slave-hllj krb5kdc[18951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:19:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:19:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:19:01.041783 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:35573
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:35573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:01.374480 18964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:01.374809 18964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:01.374953 18964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:01.382279 18964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:01.382421 18964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:01.382509 18964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:01.382588 18964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:01.382648 18964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:01.392937 18964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:35573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:35573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:01.394378 18964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:01.395879 18964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:01.408126 18969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:01.408070 18970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:01.408049 18972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:01.409170 18964 server_base.cc:1034] running on GCE node
I20260926 02:19:01.410035 18964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:01.411566 18964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:01.412797 18964 hybrid_clock.cc:648] HybridClock initialized: now 1790389141412748 us; error 37 us; skew 500 ppm
I20260926 02:19:01.413213 18964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:01 dist-test-slave-hllj krb5kdc[18951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389141, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:01.416318 18964 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:19:01.417879 18964 webserver.cc:460] Webserver started at http://127.28.152.190:36855/ using document root <none> and password file <none>
I20260926 02:19:01.418592 18964 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:01.418722 18964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:01.419013 18964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:01.422191 18964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "73e7072e26e94888852710018d7d79b8"
format_stamp: "Formatted at 2026-09-26 02:19:01 on dist-test-slave-hllj"
server_key: "46467d5e0ce0942542b118effa6a5cd6"
server_key_iv: "6e0f0ab90e18335437f3602235692eae"
server_key_version: "encryptionkey@0"
I20260926 02:19:01.422964 18964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "73e7072e26e94888852710018d7d79b8"
format_stamp: "Formatted at 2026-09-26 02:19:01 on dist-test-slave-hllj"
server_key: "46467d5e0ce0942542b118effa6a5cd6"
server_key_iv: "6e0f0ab90e18335437f3602235692eae"
server_key_version: "encryptionkey@0"
I20260926 02:19:01.428366 18964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 02:19:01.432206 18979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:01.433739 18964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:01.433928 18964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "73e7072e26e94888852710018d7d79b8"
format_stamp: "Formatted at 2026-09-26 02:19:01 on dist-test-slave-hllj"
server_key: "46467d5e0ce0942542b118effa6a5cd6"
server_key_iv: "6e0f0ab90e18335437f3602235692eae"
server_key_version: "encryptionkey@0"
I20260926 02:19:01.434139 18964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:01.448272 18964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:01.452970 18964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:01.453298 18964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:01.471415 18964 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:35573
I20260926 02:19:01.471454 19030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:35573 every 8 connection(s)
I20260926 02:19:01.472782 18964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:19:01.476660 19031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:01.477155 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 18964
I20260926 02:19:01.477330 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:19:01.477898 29282 external_mini_cluster.cc:1428] Setting key 6c6c577426cabe0f689b32c5d04076fc
I20260926 02:19:01.491325 19031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8: Bootstrap starting.
Sep 26 02:19:01 dist-test-slave-hllj krb5kdc[18951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:19:01.495002 19031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:01.496446 19031 log.cc:824] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:01.500097 19031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8: No bootstrap required, opened a new log
I20260926 02:19:01.503942 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:01.479829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33887 (local address 127.28.152.190:35573)
0926 02:19:01.480698 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:01.480719 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:01.481101 (+ 382us) server_negotiation.cc:407] Connection header received
0926 02:19:01.483059 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:01.483098 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:01.483662 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:01.484138 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:01.485497 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:01.487625 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:01.489012 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:01.490191 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:01.490375 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:01.494761 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:01.494774 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:01.494796 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:01.494851 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:01.498834 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:01.499657 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:01.499666 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:01.499675 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:01.499923 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:01.500411 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:01.500416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:01.500419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:01.501235 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:01.501551 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:01.501961 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 02:19:01.502466 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":338,"threads_started":1}
I20260926 02:19:01.508173 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:19:01.509716 19031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e7072e26e94888852710018d7d79b8" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 35573 } }
I20260926 02:19:01.510106 19031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:01.510203 19031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e7072e26e94888852710018d7d79b8, State: Initialized, Role: FOLLOWER
I20260926 02:19:01.510851 19031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e7072e26e94888852710018d7d79b8" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 35573 } }
I20260926 02:19:01.511168 19031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:19:01.511303 19031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:19:01.511463 19031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:01.514243 19031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e7072e26e94888852710018d7d79b8" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 35573 } }
I20260926 02:19:01.514767 19031 leader_election.cc:302] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73e7072e26e94888852710018d7d79b8; no voters:
I20260926 02:19:01.515377 19031 leader_election.cc:288] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:19:01.515496 19041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:01.517421 19041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [term 1 LEADER]: Becoming Leader. State: Replica: 73e7072e26e94888852710018d7d79b8, State: Running, Role: LEADER
I20260926 02:19:01.517527 19031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:01.518069 19041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e7072e26e94888852710018d7d79b8" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 35573 } }
I20260926 02:19:01.521306 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73e7072e26e94888852710018d7d79b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e7072e26e94888852710018d7d79b8" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 35573 } } }
I20260926 02:19:01.521694 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:01.522156 19043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73e7072e26e94888852710018d7d79b8. Latest consensus state: current_term: 1 leader_uuid: "73e7072e26e94888852710018d7d79b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e7072e26e94888852710018d7d79b8" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 35573 } } }
I20260926 02:19:01.522457 19043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:01.525866 19047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:19:01.530167 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:01.512632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49063 (local address 127.28.152.190:35573)
0926 02:19:01.512830 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:01.512835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:01.513277 (+ 442us) server_negotiation.cc:407] Connection header received
0926 02:19:01.513427 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:01.513433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:01.513522 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:01.513627 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:01.515087 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:01.516261 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:01.518184 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:01.518917 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:01.519069 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:01.522257 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:01.522265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:01.522270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:01.522319 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:01.526508 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:01.527548 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:01.527554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:01.527558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:01.527744 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:01.528282 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:01.528289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:01.528293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:01.528796 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:01.529052 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:01.529528 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 02:19:01.529804 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:19:01.532802 19047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:19:01.543205 19047 catalog_manager.cc:1347] Generated new cluster ID: b3aac490e71f4711b2c384b9dc6cc8ba
I20260926 02:19:01.543335 19047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:19:01.562865 19047 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:19:01.564308 19047 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:19:01.575357 19047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73e7072e26e94888852710018d7d79b8: Generated new TSK 0
I20260926 02:19:01.576452 19047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:19:01.589262 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 18964
I20260926 02:19:01.590461 19026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:01.681674 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 18964
2026-09-26T02:19:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:19:01 dist-test-slave-hllj krb5kdc[19063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:19:01 dist-test-slave-hllj krb5kdc[19063](info): set up 2 sockets
Sep 26 02:19:01 dist-test-slave-hllj krb5kdc[19063](info): commencing operation
krb5kdc: starting...
W20260926 02:19:02.706789 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:19:02 dist-test-slave-hllj krb5kdc[19063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:19:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:19:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:19:02.969038 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:38643
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:43677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:38643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:03.298491 19076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:03.298815 19076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:03.298939 19076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:03.306420 19076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:03.306552 19076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:03.306607 19076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:03.306694 19076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:03.306749 19076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:03.316880 19076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:43677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:38643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:03.318292 19076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:03.319684 19076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:03.332008 19081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:03.331954 19084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:03.331930 19082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:03.332921 19076 server_base.cc:1034] running on GCE node
I20260926 02:19:03.333807 19076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:03.335440 19076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:03.336680 19076 hybrid_clock.cc:648] HybridClock initialized: now 1790389143336634 us; error 39 us; skew 500 ppm
I20260926 02:19:03.337083 19076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:03 dist-test-slave-hllj krb5kdc[19063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389143, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:03.340619 19076 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:19:03.342358 19076 webserver.cc:460] Webserver started at http://127.28.152.190:36193/ using document root <none> and password file <none>
I20260926 02:19:03.343135 19076 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:03.343251 19076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:03.343585 19076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:03.346803 19076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "9c6ff6015db54c119c481758e62d3f62"
format_stamp: "Formatted at 2026-09-26 02:19:03 on dist-test-slave-hllj"
server_key: "ff133800006f0406386a8bab4dd81bdb"
server_key_iv: "276cda676ba232720062921d764a6d3d"
server_key_version: "encryptionkey@0"
I20260926 02:19:03.347599 19076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "9c6ff6015db54c119c481758e62d3f62"
format_stamp: "Formatted at 2026-09-26 02:19:03 on dist-test-slave-hllj"
server_key: "ff133800006f0406386a8bab4dd81bdb"
server_key_iv: "276cda676ba232720062921d764a6d3d"
server_key_version: "encryptionkey@0"
I20260926 02:19:03.353189 19076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:19:03.357029 19091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:03.358560 19076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:19:03.358752 19076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "9c6ff6015db54c119c481758e62d3f62"
format_stamp: "Formatted at 2026-09-26 02:19:03 on dist-test-slave-hllj"
server_key: "ff133800006f0406386a8bab4dd81bdb"
server_key_iv: "276cda676ba232720062921d764a6d3d"
server_key_version: "encryptionkey@0"
I20260926 02:19:03.358952 19076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:03.402923 19076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:03.407418 19076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:03.407706 19076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:03.425616 19076 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:38643
I20260926 02:19:03.425627 19142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:38643 every 8 connection(s)
I20260926 02:19:03.427017 19076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:19:03.430855 19143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:03.434259 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 19076
I20260926 02:19:03.434445 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:19:03.434974 29282 external_mini_cluster.cc:1428] Setting key d539122a2a452e2c1240a18167f231f1
I20260926 02:19:03.443384 19143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62: Bootstrap starting.
I20260926 02:19:03.447289 19143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:03.448513 19143 log.cc:824] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:03.448974 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:03.436845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41537 (local address 127.28.152.190:38643)
0926 02:19:03.437617 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:03.437629 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:03.437673 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:19:03.439367 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:03.439391 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:03.439861 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:03.440223 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:03.440856 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:03.442269 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:03.444186 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:03.444920 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:03.445071 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:03.446463 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:03.446471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:03.446487 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:19:03.447060 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:03.447150 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:19:03.447180 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:03.447273 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:03.447549 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 02:19:03.447875 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":344,"threads_started":1}
I20260926 02:19:03.451493 19143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62: No bootstrap required, opened a new log
I20260926 02:19:03.452903 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:19:03.460076 19143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ff6015db54c119c481758e62d3f62" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38643 } }
I20260926 02:19:03.460506 19143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:03.460609 19143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6ff6015db54c119c481758e62d3f62, State: Initialized, Role: FOLLOWER
I20260926 02:19:03.461277 19143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ff6015db54c119c481758e62d3f62" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38643 } }
I20260926 02:19:03.461553 19143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:19:03.461714 19143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:19:03.461915 19143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:03.464377 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:03.457433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35353 (local address 127.28.152.190:38643)
0926 02:19:03.457592 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:03.457597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:03.457962 (+ 365us) server_negotiation.cc:407] Connection header received
0926 02:19:03.458130 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:03.458136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:03.458226 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:03.458351 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:03.458897 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:03.459903 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:03.461488 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:03.462140 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:03.462285 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:03.463494 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:03.463501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:03.463510 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:19:03.463535 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:03.463595 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:19:03.463616 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:03.463725 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:03.463953 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 02:19:03.464049 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 02:19:03.465006 19143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ff6015db54c119c481758e62d3f62" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38643 } }
I20260926 02:19:03.465695 19143 leader_election.cc:302] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c6ff6015db54c119c481758e62d3f62; no voters:
I20260926 02:19:03.466388 19143 leader_election.cc:288] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:19:03.466575 19153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:03.467204 19153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [term 1 LEADER]: Becoming Leader. State: Replica: 9c6ff6015db54c119c481758e62d3f62, State: Running, Role: LEADER
I20260926 02:19:03.467785 19153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ff6015db54c119c481758e62d3f62" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38643 } }
I20260926 02:19:03.468478 19143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:03.470607 19154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c6ff6015db54c119c481758e62d3f62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ff6015db54c119c481758e62d3f62" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38643 } } }
I20260926 02:19:03.471160 19155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c6ff6015db54c119c481758e62d3f62. Latest consensus state: current_term: 1 leader_uuid: "9c6ff6015db54c119c481758e62d3f62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6ff6015db54c119c481758e62d3f62" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 38643 } } }
I20260926 02:19:03.471602 19155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:03.473382 19160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:19:03.473433 19154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:03.479064 19160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:19:03.488394 19160 catalog_manager.cc:1347] Generated new cluster ID: c6f64583e89344d4b23fc789e4d44f85
I20260926 02:19:03.488525 19160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:19:03.518898 19160 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:19:03.520954 19160 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:19:03.528903 19160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c6ff6015db54c119c481758e62d3f62: Generated new TSK 0
I20260926 02:19:03.530148 19160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:19:03.545065 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 19076
W20260926 02:19:03.546243 19138 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41537
W20260926 02:19:03.546903 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 19076: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:19:03.547040 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 19076
2026-09-26T02:19:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1864 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:19:03 dist-test-slave-hllj krb5kdc[19174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:19:03 dist-test-slave-hllj krb5kdc[19174](info): set up 2 sockets
Sep 26 02:19:03 dist-test-slave-hllj krb5kdc[19174](info): commencing operation
krb5kdc: starting...
W20260926 02:19:04.570726 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:19:04 dist-test-slave-hllj krb5kdc[19174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:19:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:19:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:19:04.830821 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:36947
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:36947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:05.170145 19187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:05.170481 19187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:05.170634 19187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:05.177984 19187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:05.178138 19187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:05.178223 19187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:05.178304 19187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:05.178364 19187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:05.188437 19187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:36947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:05.189850 19187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:05.191303 19187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:05.203559 19193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:05.203608 19195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:05.203588 19192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:05.204511 19187 server_base.cc:1034] running on GCE node
I20260926 02:19:05.205430 19187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:05.206995 19187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:05.208218 19187 hybrid_clock.cc:648] HybridClock initialized: now 1790389145208167 us; error 45 us; skew 500 ppm
I20260926 02:19:05.208581 19187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:05 dist-test-slave-hllj krb5kdc[19174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389145, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:05.215461 19187 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:19:05.217247 19187 webserver.cc:460] Webserver started at http://127.28.152.190:36939/ using document root <none> and password file <none>
I20260926 02:19:05.217996 19187 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:05.218122 19187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:05.218437 19187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:05.221699 19187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "4dadc8ce8b1d403b8fc05c9405e8bdf1"
format_stamp: "Formatted at 2026-09-26 02:19:05 on dist-test-slave-hllj"
server_key: "e861005adb5fd6a6bc9c1a6b001c90d1"
server_key_iv: "f6bba6f93b1ffe9ca9d74c526e2cc666"
server_key_version: "encryptionkey@0"
I20260926 02:19:05.222507 19187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "4dadc8ce8b1d403b8fc05c9405e8bdf1"
format_stamp: "Formatted at 2026-09-26 02:19:05 on dist-test-slave-hllj"
server_key: "e861005adb5fd6a6bc9c1a6b001c90d1"
server_key_iv: "f6bba6f93b1ffe9ca9d74c526e2cc666"
server_key_version: "encryptionkey@0"
I20260926 02:19:05.228240 19187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:19:05.232386 19202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:05.234125 19187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:05.234319 19187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "4dadc8ce8b1d403b8fc05c9405e8bdf1"
format_stamp: "Formatted at 2026-09-26 02:19:05 on dist-test-slave-hllj"
server_key: "e861005adb5fd6a6bc9c1a6b001c90d1"
server_key_iv: "f6bba6f93b1ffe9ca9d74c526e2cc666"
server_key_version: "encryptionkey@0"
I20260926 02:19:05.234534 19187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:05.240765 19187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:05.244980 19187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:05.245299 19187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:05.263392 19187 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:36947
I20260926 02:19:05.263463 19253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:36947 every 8 connection(s)
I20260926 02:19:05.264830 19187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:19:05.265976 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 19187
I20260926 02:19:05.266152 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:19:05.266574 29282 external_mini_cluster.cc:1428] Setting key c24b2a70f175fc8c96b630412a36bafb
I20260926 02:19:05.269467 19254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:05.274298 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:05.268430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56313 (local address 127.28.152.190:36947)
0926 02:19:05.269245 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:05.269258 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:05.269301 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:19:05.270954 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:05.270977 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:05.271315 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:05.271699 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:05.272677 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:05.272683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:05.272698 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:19:05.272711 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:05.272789 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:19:05.272808 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:05.273013 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:05.273168 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 02:19:05.273357 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":320,"threads_started":1}
I20260926 02:19:05.278105 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:19:05.280803 19254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1: Bootstrap starting.
I20260926 02:19:05.284469 19254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:05.284633 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:05.282228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47865 (local address 127.28.152.190:36947)
0926 02:19:05.282437 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:05.282442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:05.282847 (+ 405us) server_negotiation.cc:407] Connection header received
0926 02:19:05.282984 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:05.282989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:05.283051 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:05.283146 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:05.284009 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:05.284015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:05.284019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:19:05.284038 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:05.284088 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:19:05.284109 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:05.284222 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:05.284267 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 02:19:05.284323 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:19:05.285771 19254 log.cc:824] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:05.288651 19254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1: No bootstrap required, opened a new log
I20260926 02:19:05.296674 19254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dadc8ce8b1d403b8fc05c9405e8bdf1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36947 } }
I20260926 02:19:05.297080 19254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:05.297197 19254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dadc8ce8b1d403b8fc05c9405e8bdf1, State: Initialized, Role: FOLLOWER
I20260926 02:19:05.297783 19254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dadc8ce8b1d403b8fc05c9405e8bdf1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36947 } }
I20260926 02:19:05.297993 19254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:19:05.298127 19254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:19:05.298300 19254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:05.300616 19254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dadc8ce8b1d403b8fc05c9405e8bdf1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36947 } }
I20260926 02:19:05.301095 19254 leader_election.cc:302] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dadc8ce8b1d403b8fc05c9405e8bdf1; no voters:
I20260926 02:19:05.301683 19254 leader_election.cc:288] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:19:05.301828 19264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:05.302480 19264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [term 1 LEADER]: Becoming Leader. State: Replica: 4dadc8ce8b1d403b8fc05c9405e8bdf1, State: Running, Role: LEADER
I20260926 02:19:05.303059 19264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dadc8ce8b1d403b8fc05c9405e8bdf1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36947 } }
I20260926 02:19:05.303766 19254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:05.306051 19265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dadc8ce8b1d403b8fc05c9405e8bdf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dadc8ce8b1d403b8fc05c9405e8bdf1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36947 } } }
I20260926 02:19:05.306475 19265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:05.306890 19266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dadc8ce8b1d403b8fc05c9405e8bdf1. Latest consensus state: current_term: 1 leader_uuid: "4dadc8ce8b1d403b8fc05c9405e8bdf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dadc8ce8b1d403b8fc05c9405e8bdf1" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 36947 } } }
I20260926 02:19:05.307237 19266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:05.308887 19271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:19:05.314350 19271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:19:05.324054 19271 catalog_manager.cc:1347] Generated new cluster ID: 7c5b6d4d762b415ebf583569335823fa
I20260926 02:19:05.324170 19271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:19:05.346091 19271 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:19:05.346601 19271 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:19:05.366544 19271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dadc8ce8b1d403b8fc05c9405e8bdf1: Generated new TSK 0
I20260926 02:19:05.367717 19271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:19:05.376282 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 19187
W20260926 02:19:05.377936 19249 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56313
W20260926 02:19:05.378727 29282 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 19187: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:19:05.378875 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 19187
2026-09-26T02:19:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1832 ms)
[----------] 18 tests from AuthTokenIssuingTest (33707 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:19:05 dist-test-slave-hllj krb5kdc[19285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:19:05 dist-test-slave-hllj krb5kdc[19285](info): set up 2 sockets
Sep 26 02:19:05 dist-test-slave-hllj krb5kdc[19285](info): commencing operation
krb5kdc: starting...
W20260926 02:19:06.402830 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:19:06 dist-test-slave-hllj krb5kdc[19285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:19:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:19:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:19:06.666190 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:45119
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:45119,127.28.152.189:34317,127.28.152.188:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:07.005805 19298 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:19:07.006234 19298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:07.006341 19298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:07.006461 19298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:07.014094 19298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:07.014213 19298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:07.014298 19298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:07.014366 19298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:07.014456 19298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:07.025404 19298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:45119,127.28.152.189:34317,127.28.152.188:36411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:07.026824 19298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:07.028239 19298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:07.040443 19306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:07.040627 19303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:07.040441 19304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:07.041268 19298 server_base.cc:1034] running on GCE node
I20260926 02:19:07.042148 19298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:07.043640 19298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:07.044873 19298 hybrid_clock.cc:648] HybridClock initialized: now 1790389147044811 us; error 41 us; skew 500 ppm
I20260926 02:19:07.045238 19298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:07 dist-test-slave-hllj krb5kdc[19285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:07.048326 19298 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:19:07.049890 19298 webserver.cc:460] Webserver started at http://127.28.152.190:42285/ using document root <none> and password file <none>
I20260926 02:19:07.050616 19298 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:07.050704 19298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:07.051013 19298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:07.054101 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "9c55747495074de0b87393a8af08280d"
format_stamp: "Formatted at 2026-09-26 02:19:07 on dist-test-slave-hllj"
server_key: "43865b11c02d0a010721c7558962f701"
server_key_iv: "d5e1c6b888b4b5ee2c66262ea89b62ef"
server_key_version: "encryptionkey@0"
I20260926 02:19:07.054895 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "9c55747495074de0b87393a8af08280d"
format_stamp: "Formatted at 2026-09-26 02:19:07 on dist-test-slave-hllj"
server_key: "43865b11c02d0a010721c7558962f701"
server_key_iv: "d5e1c6b888b4b5ee2c66262ea89b62ef"
server_key_version: "encryptionkey@0"
I20260926 02:19:07.060381 19298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:19:07.064394 19313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:07.065977 19298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:07.066179 19298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "9c55747495074de0b87393a8af08280d"
format_stamp: "Formatted at 2026-09-26 02:19:07 on dist-test-slave-hllj"
server_key: "43865b11c02d0a010721c7558962f701"
server_key_iv: "d5e1c6b888b4b5ee2c66262ea89b62ef"
server_key_version: "encryptionkey@0"
I20260926 02:19:07.066391 19298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:07.087057 19298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:07.091799 19298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:07.092093 19298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:07.109993 19298 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:45119
I20260926 02:19:07.110004 19364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:45119 every 8 connection(s)
I20260926 02:19:07.111413 19298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:19:07.111862 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 19298
I20260926 02:19:07.112077 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:19:07.112905 29282 external_mini_cluster.cc:1428] Setting key 69ac713bea07202b2d0bed7fa348dd2b
I20260926 02:19:07.115712 19365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:07.122735 19365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.152.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.189@KRBTEST.COM" created.
Sep 26 02:19:07 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:19:07.146670 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.126367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45119 (local address 127.0.0.1:35078)
0926 02:19:07.127596 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.127633 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:07.129530 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:07.130503 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:07.130510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:07.131284 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:07.131668 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:07.131688 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.133315 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.133319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:07.134448 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:07.134460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.135497 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.135505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:07.135782 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:07.136437 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:07.136469 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:07.138981 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:07.142236 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:07.142250 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:07.142264 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:07.142750 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:07.143202 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:07.143207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:07.143211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:07.143384 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:07.144193 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:07.144201 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:07.144422 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 02:19:07.144651 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 02:19:07.145286 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":129,"spinlock_wait_cycles":49920,"thread_start_us":733,"threads_started":1}
I20260926 02:19:07.146708 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.126364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.28.152.190:45119)
0926 02:19:07.127263 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:07.127284 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:07.127767 (+ 483us) server_negotiation.cc:407] Connection header received
0926 02:19:07.129956 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:07.129963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:07.130254 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:07.130468 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:07.131879 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.133172 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.134611 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.135248 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.135453 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:07.139240 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:07.139249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:07.139264 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:07.139307 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:07.142088 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:07.142893 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:07.142901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:07.142903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:07.143090 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:07.143489 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:07.143493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:07.143495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:07.144053 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:07.144236 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:07.144798 (+ 562us) server_negotiation.cc:299] Negotiation successful
0926 02:19:07.145158 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":655,"thread_start_us":382,"threads_started":1}
I20260926 02:19:07.150130 19365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:19:07.151340 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.150860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42830)
0926 02:19:07.151060 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.151193 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 02:19:07.151644 19314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
I20260926 02:19:07.153303 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.152860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42836)
0926 02:19:07.153045 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.153146 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 02:19:07.153527 19365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:34317: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab.
Entry for principal kudu/127.28.152.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab.
I20260926 02:19:07.169517 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.189:34317
--webserver_interface=127.28.152.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab
--principal=kudu/127.28.152.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:45119,127.28.152.189:34317,127.28.152.188:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:19:07.202881 19365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } attempt: 1
I20260926 02:19:07.204102 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.203653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42846)
0926 02:19:07.203851 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.203971 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 02:19:07.205107 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.204772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42848)
0926 02:19:07.204891 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.204962 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 02:19:07.205345 19365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:34317: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
I20260926 02:19:07.273572 19365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } attempt: 2
I20260926 02:19:07.274710 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.274247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42860)
0926 02:19:07.274476 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.274577 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 02:19:07.275667 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.275297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42876)
0926 02:19:07.275457 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.275533 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 02:19:07.275846 19365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:34317: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
I20260926 02:19:07.367086 19365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } attempt: 3
I20260926 02:19:07.368206 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.367784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42886)
0926 02:19:07.367973 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.368070 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 02:19:07.369311 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.368859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42896)
0926 02:19:07.369012 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.369091 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 02:19:07.369546 19365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:34317: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
W20260926 02:19:07.497772 19372 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:19:07.498112 19372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:07.498186 19372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:07.498270 19372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:07.505570 19372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:07.505709 19372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:07.505769 19372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:07.505818 19372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:07.505877 19372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:07.512778 19365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } attempt: 4
I20260926 02:19:07.513988 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.513458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42910)
0926 02:19:07.513759 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.513850 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260926 02:19:07.514952 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.514600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42924)
0926 02:19:07.514745 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.514826 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 02:19:07.515139 19365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:34317: Network error: Client connection negotiation failed: client connection to 127.28.152.189:34317: connect: Connection refused (error 111)
I20260926 02:19:07.516250 19372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:45119,127.28.152.189:34317,127.28.152.188:36411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab
--principal=kudu/127.28.152.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.189:34317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.152.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:07.517611 19372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:07.518947 19372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:07.531011 19378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:07.531034 19377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:07.530992 19380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:07.531855 19372 server_base.cc:1034] running on GCE node
I20260926 02:19:07.532615 19372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:07.534039 19372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:07.535266 19372 hybrid_clock.cc:648] HybridClock initialized: now 1790389147535200 us; error 43 us; skew 500 ppm
I20260926 02:19:07.535602 19372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:07 dist-test-slave-hllj krb5kdc[19285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:07.538743 19372 init.cc:377] Logged in from keytab as kudu/127.28.152.189@KRBTEST.COM (short username kudu)
I20260926 02:19:07.540241 19372 webserver.cc:460] Webserver started at http://127.28.152.189:33345/ using document root <none> and password file <none>
I20260926 02:19:07.540943 19372 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:07.541043 19372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:07.541394 19372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:07.544363 19372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/data/instance:
uuid: "31c3c3724ea54409b82a5ae9a71673d1"
format_stamp: "Formatted at 2026-09-26 02:19:07 on dist-test-slave-hllj"
server_key: "fe120e7b6ae6efb906fe0ebc9344002b"
server_key_iv: "d8b7d3e4688dc30ca3a7f540233aabc6"
server_key_version: "encryptionkey@0"
I20260926 02:19:07.545173 19372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/wal/instance:
uuid: "31c3c3724ea54409b82a5ae9a71673d1"
format_stamp: "Formatted at 2026-09-26 02:19:07 on dist-test-slave-hllj"
server_key: "fe120e7b6ae6efb906fe0ebc9344002b"
server_key_iv: "d8b7d3e4688dc30ca3a7f540233aabc6"
server_key_version: "encryptionkey@0"
I20260926 02:19:07.550236 19372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:19:07.553839 19387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:07.555382 19372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:19:07.555560 19372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/wal
uuid: "31c3c3724ea54409b82a5ae9a71673d1"
format_stamp: "Formatted at 2026-09-26 02:19:07 on dist-test-slave-hllj"
server_key: "fe120e7b6ae6efb906fe0ebc9344002b"
server_key_iv: "d8b7d3e4688dc30ca3a7f540233aabc6"
server_key_version: "encryptionkey@0"
I20260926 02:19:07.555776 19372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:07.579780 19372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:07.584240 19372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:07.584609 19372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:07.603554 19372 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.189:34317
I20260926 02:19:07.603579 19438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.189:34317 every 8 connection(s)
I20260926 02:19:07.605085 19372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
I20260926 02:19:07.609097 19439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:07.614348 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 19372
I20260926 02:19:07.614533 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-1/wal/instance
I20260926 02:19:07.614984 29282 external_mini_cluster.cc:1428] Setting key d438245140ccc5932cd42496b96e2a01
I20260926 02:19:07.615993 19439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:19:07 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:19:07.636591 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.618859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.28.152.190:45119)
0926 02:19:07.619038 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:07.619043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:07.620588 (+ 1545us) server_negotiation.cc:407] Connection header received
0926 02:19:07.622634 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:07.622641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:07.622724 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:07.622813 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:07.624446 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.625157 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.626711 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.627131 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.627248 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:07.630957 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:07.630963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:07.630966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:07.631008 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:07.633558 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:07.634307 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:07.634313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:07.634316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:07.634438 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:07.634838 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:07.634842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:07.634846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:07.635135 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:07.635288 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:07.636039 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 02:19:07.636273 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.28.152.188@KRBTEST.COM; defaulting to no policy
I20260926 02:19:07.637677 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.619594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45119 (local address 127.0.0.1:35090)
0926 02:19:07.620393 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.620449 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:07.622316 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:07.623087 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:07.623097 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:07.623657 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:07.624261 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:07.624279 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.625310 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.625315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:07.626554 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:07.626566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.627370 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.627378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:07.627617 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:07.628306 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:07.628332 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:07.630707 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:07.633700 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:07.633711 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:07.633730 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:07.634188 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:07.634544 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:07.634550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:07.634554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:07.634735 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:07.635255 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:07.635264 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:07.635668 (+ 404us) client_negotiation.cc:769] Sending connection context
0926 02:19:07.635915 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 02:19:07.636454 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":328,"threads_started":1}
I20260926 02:19:07.639394 19439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.188@KRBTEST.COM" created.
Sep 26 02:19:07 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:19:07.655297 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.640183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42934)
0926 02:19:07.640368 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.640392 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:07.640533 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:07.641566 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:07.641571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:07.642007 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:07.642179 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:07.642190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.643546 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.643551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:07.644660 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:07.644673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.645600 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.645607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:07.645779 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:07.646355 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:07.646376 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:07.648744 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:07.652858 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:07.652862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:07.652865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:07.653310 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:07.653728 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:07.653732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:07.653734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:07.653892 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:07.654574 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:07.654578 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:07.654694 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 02:19:07.654799 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 02:19:07.655022 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 02:19:07.655640 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.640450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.28.152.189:34317)
0926 02:19:07.641092 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:07.641098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:07.641123 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:19:07.641219 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:07.641225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:07.641417 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:07.641530 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:07.642346 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.643416 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.644808 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.645452 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.645555 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:07.649004 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:07.649023 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:07.649045 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:07.649092 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:07.652677 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:07.653445 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:07.653455 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:07.653458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:07.653608 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:07.653987 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:07.653991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:07.653993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:07.654451 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:07.654644 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:07.654977 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 02:19:07.655240 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":358,"threads_started":1}
I20260926 02:19:07.658190 19439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:19:07.659389 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.658880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48270)
0926 02:19:07.659062 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.659226 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260926 02:19:07.659623 19391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
I20260926 02:19:07.661247 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.660840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48284)
0926 02:19:07.661013 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.661099 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 02:19:07.661440 19439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36411: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab.
Entry for principal kudu/127.28.152.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab.
I20260926 02:19:07.678419 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.188:36411
--webserver_interface=127.28.152.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab
--principal=kudu/127.28.152.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:45119,127.28.152.189:34317,127.28.152.188:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:19:07.710763 19439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } attempt: 1
I20260926 02:19:07.711903 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.711392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48300)
0926 02:19:07.711626 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.711744 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 02:19:07.712850 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.712503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48316)
0926 02:19:07.712660 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.712736 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 02:19:07.713047 19439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36411: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
I20260926 02:19:07.781296 19439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } attempt: 2
I20260926 02:19:07.782648 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.782128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48322)
0926 02:19:07.782342 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.782440 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 02:19:07.783684 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.783307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48336)
0926 02:19:07.783474 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.783553 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:19:07.783895 19439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36411: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
I20260926 02:19:07.814388 19365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } attempt: 5
Sep 26 02:19:07 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:19:07.829085 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.815140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42942)
0926 02:19:07.815379 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.815401 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:07.815536 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:07.815938 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:07.815941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:07.816294 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:07.816536 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:07.816545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.817958 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.817963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:07.818835 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:07.818848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.819900 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.819905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:07.820072 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:07.820683 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:07.820702 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:07.822798 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:07.826694 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:07.826698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:07.826701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:07.827104 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:07.827526 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:07.827531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:07.827534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:07.827684 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:07.828313 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:07.828317 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:07.828465 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 02:19:07.828579 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:19:07.828822 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 02:19:07.829370 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.815395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.28.152.189:34317)
0926 02:19:07.815605 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:07.815612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:07.815633 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:19:07.815698 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:07.815703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:07.815796 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:07.815909 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:07.816707 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.817821 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.819003 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:07.819737 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:07.819866 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:07.823024 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:07.823031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:07.823036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:07.823078 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:07.826512 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:07.827234 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:07.827240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:07.827244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:07.827402 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:07.827791 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:07.827796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:07.827799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:07.828181 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:07.828344 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:07.828721 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 02:19:07.829005 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:19:07.830404 19365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:19:07.831538 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.831103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48352)
0926 02:19:07.831297 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.831389 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 02:19:07.832530 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.832178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48354)
0926 02:19:07.832355 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.832431 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 02:19:07.832721 19365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36411: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
I20260926 02:19:07.875118 19439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } attempt: 3
I20260926 02:19:07.876268 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.875825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48356)
0926 02:19:07.876016 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.876108 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 02:19:07.877311 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.876872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48358)
0926 02:19:07.877077 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.877194 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260926 02:19:07.877493 19439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36411: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
I20260926 02:19:07.883936 19365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } attempt: 1
I20260926 02:19:07.885510 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.884595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48360)
0926 02:19:07.885113 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.885286 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
I20260926 02:19:07.886879 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.886433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48366)
0926 02:19:07.886610 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.886714 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 02:19:07.887236 19365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36411: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
I20260926 02:19:07.955499 19365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } attempt: 2
I20260926 02:19:07.956719 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.956141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48368)
0926 02:19:07.956400 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.956517 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:19:07.957829 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:07.957422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48378)
0926 02:19:07.957571 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:07.957664 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 02:19:07.958097 19365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36411: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
I20260926 02:19:08.020709 19439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } attempt: 4
I20260926 02:19:08.021911 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.021419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48388)
0926 02:19:08.021650 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:08.021738 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 02:19:08.022887 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.022529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48392)
0926 02:19:08.022704 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:08.022775 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 02:19:08.023098 19439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36411: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
W20260926 02:19:08.023483 19446 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:19:08.023836 19446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:08.023928 19446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:08.024037 19446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:08.031553 19446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:08.031672 19446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:08.031754 19446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:08.031832 19446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:08.031919 19446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:08.042392 19446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:45119,127.28.152.189:34317,127.28.152.188:36411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab
--principal=kudu/127.28.152.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.188:36411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.152.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:08.043764 19446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:08.045187 19446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:08.057111 19451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:08.057271 19452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:08.057279 19454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:08.058118 19446 server_base.cc:1034] running on GCE node
I20260926 02:19:08.058949 19446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:08.060567 19446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:08.061844 19446 hybrid_clock.cc:648] HybridClock initialized: now 1790389148061728 us; error 43 us; skew 500 ppm
I20260926 02:19:08.062260 19446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:19:08.064313 19365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } attempt: 3
Sep 26 02:19:08 dist-test-slave-hllj krb5kdc[19285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:08.065485 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.064924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48402)
0926 02:19:08.065114 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:08.065239 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 02:19:08.065780 19446 init.cc:377] Logged in from keytab as kudu/127.28.152.188@KRBTEST.COM (short username kudu)
I20260926 02:19:08.066679 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.066204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48412)
0926 02:19:08.066481 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:08.066563 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260926 02:19:08.066879 19365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36411: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36411: connect: Connection refused (error 111)
I20260926 02:19:08.067422 19446 webserver.cc:460] Webserver started at http://127.28.152.188:33141/ using document root <none> and password file <none>
I20260926 02:19:08.068198 19446 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:08.068289 19446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:08.068557 19446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:08.071734 19446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/data/instance:
uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3"
format_stamp: "Formatted at 2026-09-26 02:19:08 on dist-test-slave-hllj"
server_key: "e2915065d1418edb804ea59444660d32"
server_key_iv: "9d38af98c6b14be9126b690fc14d66ef"
server_key_version: "encryptionkey@0"
I20260926 02:19:08.072535 19446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/wal/instance:
uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3"
format_stamp: "Formatted at 2026-09-26 02:19:08 on dist-test-slave-hllj"
server_key: "e2915065d1418edb804ea59444660d32"
server_key_iv: "9d38af98c6b14be9126b690fc14d66ef"
server_key_version: "encryptionkey@0"
I20260926 02:19:08.077788 19446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 02:19:08.081395 19461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:08.082930 19446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:19:08.083101 19446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/wal
uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3"
format_stamp: "Formatted at 2026-09-26 02:19:08 on dist-test-slave-hllj"
server_key: "e2915065d1418edb804ea59444660d32"
server_key_iv: "9d38af98c6b14be9126b690fc14d66ef"
server_key_version: "encryptionkey@0"
I20260926 02:19:08.083325 19446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:08.104466 19446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:08.109372 19446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:08.109642 19446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:08.128419 19446 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.188:36411
I20260926 02:19:08.128438 19512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.188:36411 every 8 connection(s)
I20260926 02:19:08.129989 19446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
I20260926 02:19:08.133189 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 19446
I20260926 02:19:08.133383 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790388791636863-29282-0/minicluster-data/master-2/wal/instance
I20260926 02:19:08.133919 29282 external_mini_cluster.cc:1428] Setting key c8bb7a4ffb6ba4f1aa648fbe6e4c2718
I20260926 02:19:08.134270 19513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:08.144373 19513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:19:08 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
Sep 26 02:19:08 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:19:08.160692 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.135864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.28.152.188:36411)
0926 02:19:08.136802 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.136815 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.136863 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:19:08.139254 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.139285 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.139811 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.140230 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.142210 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.143637 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.145576 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.146361 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.146509 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.151694 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:08.151708 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:08.151723 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:08.151780 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:08.155764 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.156640 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.156648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.156655 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.156899 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.157359 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.157365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.157368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.158000 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:08.158242 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.158945 (+ 703us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.159406 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":361,"threads_started":1}
I20260926 02:19:08.163719 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.147575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.28.152.190:45119)
0926 02:19:08.148045 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.148051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.148524 (+ 473us) server_negotiation.cc:407] Connection header received
0926 02:19:08.148700 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.148706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.148795 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.148912 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.150185 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.151241 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.153260 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.153810 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.153939 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.157727 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:08.157733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:08.157736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:08.157775 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:08.160799 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.161676 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.161680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.161683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.161826 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.162203 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.162207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.162209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.162502 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:08.162640 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.163164 (+ 524us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.163380 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":235,"threads_started":1}
I20260926 02:19:08.164052 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.147573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:45119 (local address 127.0.0.1:35102)
0926 02:19:08.148368 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:08.148421 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:08.148579 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:08.148986 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:08.148992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:08.149760 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:08.150013 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:08.150023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.151415 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.151419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:08.153080 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:08.153096 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.154030 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.154039 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:08.154244 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.154868 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:08.154896 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:08.157485 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:08.160992 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:08.161007 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:08.161015 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:08.161536 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:08.161942 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:08.161948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:08.161952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:08.162085 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:08.162637 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:08.162645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:08.162843 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 02:19:08.163010 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 02:19:08.163663 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":377,"threads_started":1}
I20260926 02:19:08.165935 19513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:19:08 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
Sep 26 02:19:08 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:19:08.180480 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.166577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:34317 (local address 127.0.0.1:42954)
0926 02:19:08.166760 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:08.166780 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:08.166882 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:08.167206 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:08.167211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:08.167667 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:08.167815 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:08.167825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.169237 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.169242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:08.170605 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:08.170620 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.171565 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.171571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:08.171706 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.172230 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:08.172248 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:08.174770 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:08.177958 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:08.177962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:08.177965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:08.178353 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:08.178763 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:08.178767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:08.178770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:08.178918 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:08.179445 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:08.179451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:08.179644 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 02:19:08.179785 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 02:19:08.180117 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 02:19:08.180552 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.166700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.28.152.189:34317)
0926 02:19:08.166879 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.166884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.166903 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:19:08.167012 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.167015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.167077 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.167156 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.168014 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.169065 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.170757 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.171393 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.171543 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.175005 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:08.175010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:08.175014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:08.175051 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:08.177810 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.178490 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.178494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.178497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.178644 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.179033 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.179037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.179039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.179310 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:08.179433 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.179927 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.180182 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 02:19:08.180567 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.163831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.28.152.190:45119)
0926 02:19:08.164314 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.164334 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.164353 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:19:08.164408 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.164414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.164515 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.164618 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.166267 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.167453 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.169225 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.169993 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.170130 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.174810 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:08.174817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:08.174822 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:08.174870 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:08.177683 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.178424 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.178430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.178434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.178573 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.178983 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.178989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.178992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.179391 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:08.179677 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.179990 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.180228 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":1594,"threads_started":1}
I20260926 02:19:08.181824 19513 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:19:08 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389148, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
Sep 26 02:19:08 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:19:08.195693 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.182835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.28.152.188:36411)
0926 02:19:08.182980 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.182985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.183000 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:19:08.183040 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.183044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.183110 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.183195 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.184087 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.184890 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.186282 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.186690 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.186807 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.190535 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:08.190540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:08.190543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:08.190574 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:08.193306 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.193894 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.193900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.193903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.194013 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.194333 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.194339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.194343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.194627 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:08.194752 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.195103 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.195303 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 02:19:08.195851 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.182662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48430)
0926 02:19:08.182786 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:08.182800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:08.182903 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:08.183248 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:08.183252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:08.183729 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:08.183923 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:08.183936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.185016 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.185020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:08.186133 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:08.186148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.186883 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.186890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:08.187125 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.187771 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:08.187789 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:08.190325 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:08.193425 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:08.193431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:08.193435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:08.193774 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:08.194108 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:08.194113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:08.194117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:08.194233 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:08.194732 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:08.194737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:08.194875 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 02:19:08.195008 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 02:19:08.195479 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":138}
I20260926 02:19:08.198412 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.182486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.28.152.189:34317)
0926 02:19:08.182651 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.182656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.183061 (+ 405us) server_negotiation.cc:407] Connection header received
0926 02:19:08.183187 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.183192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.183262 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.183352 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.184750 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.185596 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.187224 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.187811 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.188146 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.192804 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:08.192810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:08.192813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:08.192850 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:08.195435 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.196291 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.196296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.196298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.196472 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.196871 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.196876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.196880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.197246 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:08.197536 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.197833 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.198083 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 02:19:08.201333 19513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3: Bootstrap starting.
I20260926 02:19:08.203181 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:19:08.204674 19513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:08.205857 19513 log.cc:824] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:08.208738 19513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3: No bootstrap required, opened a new log
I20260926 02:19:08.205797 19340 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9c55747495074de0b87393a8af08280d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35112
I20260926 02:19:08.215915 19513 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } }
I20260926 02:19:08.216260 19513 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1df2d1a3ca4e7f94abdde16e5f44f3, State: Initialized, Role: FOLLOWER
I20260926 02:19:08.216722 19513 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } }
I20260926 02:19:08.217686 19521 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } } }
I20260926 02:19:08.218077 19521 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:08.218640 19513 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:19:08.228005 19532 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:19:08.228155 19532 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:19:08.225771 19446 master_runner.cc:186] Error getting master registration for 127.28.152.190:45119: OK, instance_id { permanent_uuid: "9c55747495074de0b87393a8af08280d" instance_seqno: 1790389147104889 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:19:08.236874 19446 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:19:08.244108 19365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } attempt: 4
Sep 26 02:19:08 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
I20260926 02:19:08.260488 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.244921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48444)
0926 02:19:08.245111 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:08.245161 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:08.245274 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:08.245646 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:08.245650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:08.246184 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:08.246399 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:08.246411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.247912 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.247917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:08.249460 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:08.249476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.250525 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.250532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:08.250729 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.251462 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:08.251486 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:08.253702 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:08.257728 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:08.257735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:08.257739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:08.258160 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:08.258588 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:08.258594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:08.258598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:08.258753 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:08.259458 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:08.259463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:08.259639 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 02:19:08.259784 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:19:08.260100 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 02:19:08.260491 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.245034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.28.152.188:36411)
0926 02:19:08.245292 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.245298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.245327 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:19:08.245394 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.245400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.245500 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.245609 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.246577 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.247756 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.249616 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.250364 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.250504 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.253929 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:08.253935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:08.253939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:08.253975 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:08.257575 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.258317 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.258324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.258326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.258469 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.258880 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.258886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.258889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.259312 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:08.259480 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.259916 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.260122 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:19:08.266716 19365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d: Bootstrap starting.
I20260926 02:19:08.271139 19365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:08.272586 19365 log.cc:824] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:08.276402 19365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d: No bootstrap required, opened a new log
I20260926 02:19:08.284202 19365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } }
I20260926 02:19:08.284874 19365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c55747495074de0b87393a8af08280d, State: Initialized, Role: FOLLOWER
I20260926 02:19:08.285526 19365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } }
I20260926 02:19:08.286649 19535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } } }
I20260926 02:19:08.287070 19535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:08.288213 19365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:08.288323 19340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 02:19:08.288528 19340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:08.291128 19340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } }
I20260926 02:19:08.292689 19340 leader_election.cc:288] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [CANDIDATE]: Term 1 election: Requested vote from peers 31c3c3724ea54409b82a5ae9a71673d1 (127.28.152.189:34317), fa1df2d1a3ca4e7f94abdde16e5f44f3 (127.28.152.188:36411)
I20260926 02:19:08.294170 19488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c55747495074de0b87393a8af08280d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3"
I20260926 02:19:08.294667 19488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:08.298714 19488 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c55747495074de0b87393a8af08280d in term 1.
I20260926 02:19:08.299595 19317 leader_election.cc:302] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c55747495074de0b87393a8af08280d, fa1df2d1a3ca4e7f94abdde16e5f44f3; no voters:
I20260926 02:19:08.300252 19535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:08.311493 19535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [term 1 LEADER]: Becoming Leader. State: Replica: 9c55747495074de0b87393a8af08280d, State: Running, Role: LEADER
I20260926 02:19:08.312389 19535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } }
I20260926 02:19:08.319861 19542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c55747495074de0b87393a8af08280d. Latest consensus state: current_term: 1 leader_uuid: "9c55747495074de0b87393a8af08280d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } } }
I20260926 02:19:08.320549 19542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:08.322280 19439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } attempt: 5
I20260926 02:19:08.325923 19547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 02:19:08 dist-test-slave-hllj krb5kdc[19285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389147, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
I20260926 02:19:08.341204 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.322951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36411 (local address 127.0.0.1:48446)
0926 02:19:08.323337 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:08.323357 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:08.323483 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:08.324571 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:08.324577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:08.325104 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:08.325360 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:08.325375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.326819 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.326826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:08.328200 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:08.328219 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.329527 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.329534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:08.329772 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.330462 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:08.330487 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:08.333425 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:08.338154 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:08.338160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:08.338165 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:08.338696 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:08.339182 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:08.339187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:08.339190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:08.339396 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:08.340101 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:08.340107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:08.340270 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 02:19:08.340413 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 02:19:08.340731 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253}
I20260926 02:19:08.342391 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.323837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48446 (local address 127.28.152.188:36411)
0926 02:19:08.324043 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.324050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.324070 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:19:08.324138 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.324144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.324257 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.324363 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.325580 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.326648 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.328411 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.329156 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.329303 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.333703 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:08.333712 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:08.333718 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:08.333769 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:08.337931 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.338850 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.338857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.338861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.339046 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.339527 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.339532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.339535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.339887 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:08.340042 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.341647 (+ 1605us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.341939 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:19:08.349026 19439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1: Bootstrap starting.
W20260926 02:19:08.349458 19319 tablet.cc:2369] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:19:08.353801 19547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:19:08.354427 19439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:08.355782 19439 log.cc:824] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1: Log is configured to *not* fsync() on all Append() calls
W20260926 02:19:08.361967 19467 tablet.cc:2369] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:19:08.367035 19439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1: No bootstrap required, opened a new log
I20260926 02:19:08.380035 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.332043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.28.152.189:34317)
0926 02:19:08.332221 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.332226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.350560 (+ 18334us) server_negotiation.cc:407] Connection header received
0926 02:19:08.351053 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.351058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.351132 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.351193 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.353514 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.354580 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.356383 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.357012 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.357121 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.368523 (+ 11402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:08.368531 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:08.368536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:08.368582 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:08.376061 (+ 7479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.376964 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.376971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.376975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.377187 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.377639 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.377645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.377649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.378038 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:08.378292 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.379357 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.379627 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:19:08.382378 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.332278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.28.152.190:45119)
0926 02:19:08.332461 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.332467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.345600 (+ 13133us) server_negotiation.cc:407] Connection header received
0926 02:19:08.346308 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.346315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.346405 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.346510 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.348762 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.349953 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.355547 (+ 5594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.356774 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.356905 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.371663 (+ 14758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:08.371670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:08.371675 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:08.371731 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:08.375456 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.378832 (+ 3376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.378839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.378843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.379075 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.379510 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.379516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.379520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.379908 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:08.380148 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.380947 (+ 799us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.381201 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:19:08.393298 19439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } }
I20260926 02:19:08.393999 19439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c3c3724ea54409b82a5ae9a71673d1, State: Initialized, Role: FOLLOWER
I20260926 02:19:08.394826 19439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } }
I20260926 02:19:08.397780 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.335824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48460 (local address 127.28.152.188:36411)
0926 02:19:08.343774 (+ 7950us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.343780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.362430 (+ 18650us) server_negotiation.cc:407] Connection header received
0926 02:19:08.362746 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.362752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.362858 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.362937 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.371671 (+ 8734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.372793 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.385301 (+ 12508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.386107 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.386226 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.390181 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:08.390190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:08.390195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:08.390245 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:08.394123 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.395169 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.395177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.395181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.395343 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:08.395806 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:08.395812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:08.395816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:08.396193 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:08.396449 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.397022 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.397326 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7217,"thread_start_us":8149,"threads_started":1}
I20260926 02:19:08.398651 19488 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [term 1 FOLLOWER]: Refusing update from remote peer 9c55747495074de0b87393a8af08280d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:08.400100 19542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:08.400857 19561 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:19:08.412456 19560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } } }
I20260926 02:19:08.413007 19560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:08.415124 19414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c55747495074de0b87393a8af08280d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "31c3c3724ea54409b82a5ae9a71673d1"
I20260926 02:19:08.415486 19413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:08.415783 19439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:08.421893 19413 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [term 1 FOLLOWER]: Refusing update from remote peer 9c55747495074de0b87393a8af08280d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:08.426105 19414 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c55747495074de0b87393a8af08280d in term 1.
I20260926 02:19:08.429678 19542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:08.408169 19298 master_runner.cc:186] Error getting master registration for 127.28.152.189:34317: OK, instance_id { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" instance_seqno: 1790389147597835 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:19:08.463459 19560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c55747495074de0b87393a8af08280d. Latest consensus state: current_term: 1 leader_uuid: "9c55747495074de0b87393a8af08280d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } } }
I20260926 02:19:08.463881 19560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:08.469666 19560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c55747495074de0b87393a8af08280d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } } }
I20260926 02:19:08.470001 19560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:08.478195 19521 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c55747495074de0b87393a8af08280d. Latest consensus state: current_term: 1 leader_uuid: "9c55747495074de0b87393a8af08280d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } } }
I20260926 02:19:08.478595 19521 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:08.467614 19557 mvcc.cc:205] Tried to move back new op lower bound from 7333433951826329600 to 7333433951495180288. Current Snapshot: MvccSnapshot[applied={T|T < 7333433951826329600}]
I20260926 02:19:08.486887 19535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c55747495074de0b87393a8af08280d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } } }
I20260926 02:19:08.487303 19535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:08.488986 19298 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:19:08.494594 19535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c55747495074de0b87393a8af08280d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } } }
I20260926 02:19:08.494961 19535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:08.497144 19547 catalog_manager.cc:1347] Generated new cluster ID: c00406c05c8d417f90f870881ed409b4
I20260926 02:19:08.497274 19547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:19:08.499769 19564 mvcc.cc:205] Tried to move back new op lower bound from 7333433951826329600 to 7333433951495180288. Current Snapshot: MvccSnapshot[applied={T|T < 7333433951826329600}]
I20260926 02:19:08.505582 19563 mvcc.cc:205] Tried to move back new op lower bound from 7333433951826329600 to 7333433951495180288. Current Snapshot: MvccSnapshot[applied={T|T < 7333433951826329600}]
I20260926 02:19:08.507548 19521 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c55747495074de0b87393a8af08280d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c55747495074de0b87393a8af08280d" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 45119 } } peers { permanent_uuid: "31c3c3724ea54409b82a5ae9a71673d1" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 34317 } } peers { permanent_uuid: "fa1df2d1a3ca4e7f94abdde16e5f44f3" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36411 } } }
I20260926 02:19:08.507858 19521 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:08.537720 19547 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:19:08.539904 19547 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:19:08.573524 19547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c55747495074de0b87393a8af08280d: Generated new TSK 0
I20260926 02:19:08.574996 19547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:19:08.617312 19579 catalog_manager.cc:1259] Loaded cluster ID: c00406c05c8d417f90f870881ed409b4
I20260926 02:19:08.617590 19579 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1: loading cluster ID for follower catalog manager: success
I20260926 02:19:08.621739 19579 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1: acquiring CA information for follower catalog manager: success
I20260926 02:19:08.624002 19579 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31c3c3724ea54409b82a5ae9a71673d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 02:19:08.740139 19516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:08.728882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48468 (local address 127.28.152.188:36411)
0926 02:19:08.729090 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.729097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.738185 (+ 9088us) server_negotiation.cc:407] Connection header received
0926 02:19:08.738463 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.738469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.738558 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.738631 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.739867 (+ 1236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48468: BlockingRecv error: recv got EOF from 127.0.0.1:48468 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 02:19:08.740142 19443 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:08.715130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42976 (local address 127.28.152.189:34317)
0926 02:19:08.715334 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.715340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.737941 (+ 22601us) server_negotiation.cc:407] Connection header received
0926 02:19:08.738303 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.738311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.738423 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.738499 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.739865 (+ 1366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42976: BlockingRecv error: recv got EOF from 127.0.0.1:42976 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 02:19:08.742564 19517 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:08.713053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35130 (local address 127.28.152.190:45119)
0926 02:19:08.713309 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.713314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.732067 (+ 18753us) server_negotiation.cc:407] Connection header received
0926 02:19:08.732325 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.732331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.732423 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.732532 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.742278 (+ 9746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35130: BlockingRecv error: recv got EOF from 127.0.0.1:35130 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:19:08.810025 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.785800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.28.152.189:34317)
0926 02:19:08.786018 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.786025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.801843 (+ 15818us) server_negotiation.cc:407] Connection header received
0926 02:19:08.802142 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.802150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.802399 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.802558 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:08.803354 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.804523 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.806698 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.807412 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.807524 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.808367 (+ 843us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:08.809104 (+ 737us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:08.809283 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.809534 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.809642 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 02:19:08.854631 19516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:08.845050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.28.152.188:36411)
0926 02:19:08.845281 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.845287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.852509 (+ 7222us) server_negotiation.cc:407] Connection header received
0926 02:19:08.852906 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.852914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.853073 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.853190 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.854378 (+ 1188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48478: BlockingRecv error: recv got EOF from 127.0.0.1:48478 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 02:19:08.886672 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.879566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.28.152.189:34317)
0926 02:19:08.879780 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.879787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.879813 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:19:08.880063 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.880069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.880186 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.880265 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:08.881051 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.882233 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.884313 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.885003 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.885195 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.885692 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:08.885868 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:08.885969 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.886143 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.886239 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 02:19:08.910126 19516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:08.907292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.28.152.188:36411)
0926 02:19:08.907476 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.907483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.908069 (+ 586us) server_negotiation.cc:407] Connection header received
0926 02:19:08.908390 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.908397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.908494 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.908566 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.909884 (+ 1318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48484: BlockingRecv error: recv got EOF from 127.0.0.1:48484 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:19:08.925175 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.917493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.28.152.189:34317)
0926 02:19:08.917694 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.917701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.918232 (+ 531us) server_negotiation.cc:407] Connection header received
0926 02:19:08.918571 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.918578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.918676 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.918743 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:08.919541 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.920732 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.922830 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.923519 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.923620 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.924228 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:08.924424 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:08.924497 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.924718 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.924806 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 02:19:08.937896 19516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:08.935267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.28.152.188:36411)
0926 02:19:08.935482 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.935488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.936060 (+ 572us) server_negotiation.cc:407] Connection header received
0926 02:19:08.936376 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.936383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.936480 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.936554 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.937655 (+ 1101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48498: BlockingRecv error: recv got EOF from 127.0.0.1:48498 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:19:08.952760 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.946379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.28.152.189:34317)
0926 02:19:08.946629 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.946636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.947029 (+ 393us) server_negotiation.cc:407] Connection header received
0926 02:19:08.947225 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.947229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.947305 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.947401 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:08.947874 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.948679 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.950861 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.951429 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.951552 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.952090 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:08.952212 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:08.952277 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.952434 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.952514 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260926 02:19:08.961490 19516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:08.959363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.28.152.188:36411)
0926 02:19:08.959697 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.959704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.959995 (+ 291us) server_negotiation.cc:407] Connection header received
0926 02:19:08.960192 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.960197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.960312 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.960399 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.961283 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48512: BlockingRecv error: recv got EOF from 127.0.0.1:48512 (error 108)
Metrics: {"server-negotiator.queue_time_us":201}
I20260926 02:19:08.978986 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:08.973057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.28.152.189:34317)
0926 02:19:08.973311 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.973318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.973664 (+ 346us) server_negotiation.cc:407] Connection header received
0926 02:19:08.973837 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.973844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.973938 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.974048 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:08.974418 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.975509 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.977260 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:08.977754 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:08.977883 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:08.978281 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:08.978405 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:08.978479 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:08.978622 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 02:19:08.978702 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260926 02:19:08.986670 19516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:08.984752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.28.152.188:36411)
0926 02:19:08.984948 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:08.984955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:08.985259 (+ 304us) server_negotiation.cc:407] Connection header received
0926 02:19:08.985460 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:08.985466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:08.985545 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:08.985630 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:08.986430 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48518: BlockingRecv error: recv got EOF from 127.0.0.1:48518 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:19:09.013374 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:09.006722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.28.152.189:34317)
0926 02:19:09.006940 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:09.006947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:09.007344 (+ 397us) server_negotiation.cc:407] Connection header received
0926 02:19:09.007587 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:09.007592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:09.007664 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:09.007753 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:09.008336 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.009280 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.011325 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.011813 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.011918 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:09.012628 (+ 710us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:09.012754 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:09.012825 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:09.013017 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 02:19:09.013097 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260926 02:19:09.021718 19516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:09.019761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.28.152.188:36411)
0926 02:19:09.019965 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:09.019993 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:09.020224 (+ 231us) server_negotiation.cc:407] Connection header received
0926 02:19:09.020446 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:09.020453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:09.020549 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:09.020631 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:09.021499 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48532: BlockingRecv error: recv got EOF from 127.0.0.1:48532 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:19:09.064090 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:09.057274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.28.152.189:34317)
0926 02:19:09.057494 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:09.057501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:09.057872 (+ 371us) server_negotiation.cc:407] Connection header received
0926 02:19:09.058238 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:09.058245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:09.058374 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:09.058496 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:09.059028 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.060492 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.062070 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.062729 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.062907 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:09.063279 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:09.063435 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:09.063535 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:09.063651 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 02:19:09.063747 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 02:19:09.072458 19516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:09.070224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.28.152.188:36411)
0926 02:19:09.070439 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:09.070445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:09.070964 (+ 519us) server_negotiation.cc:407] Connection header received
0926 02:19:09.071175 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:09.071179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:09.071256 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:09.071341 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:09.072201 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48544: BlockingRecv error: recv got EOF from 127.0.0.1:48544 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:19:09.146610 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:09.140343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.28.152.189:34317)
0926 02:19:09.140559 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:09.140566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:09.141040 (+ 474us) server_negotiation.cc:407] Connection header received
0926 02:19:09.141257 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:09.141261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:09.141337 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:09.141433 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:09.141879 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.142642 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.144706 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.145221 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.145324 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:09.145915 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:09.146035 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:09.146117 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:09.146260 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 02:19:09.146338 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 02:19:09.154426 19516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:09.152562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.28.152.188:36411)
0926 02:19:09.152803 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:09.152809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:09.153079 (+ 270us) server_negotiation.cc:407] Connection header received
0926 02:19:09.153277 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:09.153281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:09.153366 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:09.153459 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:09.154213 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48548: BlockingRecv error: recv got EOF from 127.0.0.1:48548 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 02:19:09.230369 19532 catalog_manager.cc:1259] Loaded cluster ID: c00406c05c8d417f90f870881ed409b4
I20260926 02:19:09.230484 19532 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3: loading cluster ID for follower catalog manager: success
I20260926 02:19:09.233021 19532 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3: acquiring CA information for follower catalog manager: success
I20260926 02:19:09.234959 19532 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fa1df2d1a3ca4e7f94abdde16e5f44f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:19:09.291324 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:09.286068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.28.152.189:34317)
0926 02:19:09.286362 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:09.286368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:09.286588 (+ 220us) server_negotiation.cc:407] Connection header received
0926 02:19:09.286799 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:09.286804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:09.286884 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:09.286964 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:09.287401 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.288172 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.289666 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.290063 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.290179 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:09.290583 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:09.290708 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:09.290782 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:09.290932 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 02:19:09.291026 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:19:09.305665 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:09.299246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.28.152.188:36411)
0926 02:19:09.299496 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:09.299502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:09.299770 (+ 268us) server_negotiation.cc:407] Connection header received
0926 02:19:09.299951 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:09.299955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:09.300132 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:09.300312 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:09.300702 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.301758 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.303385 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.303813 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.303917 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:09.304552 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:09.305094 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:09.305270 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:09.305355 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 02:19:09.305431 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 02:19:09.324697 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:09.316632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.28.152.188:36411)
0926 02:19:09.316840 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:09.316847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:09.318144 (+ 1297us) server_negotiation.cc:407] Connection header received
0926 02:19:09.318472 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:09.318479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:09.318582 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:09.318650 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:09.319372 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.320440 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.322609 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.323264 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.323395 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:09.323939 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:09.324078 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:09.324155 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:09.324329 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 02:19:09.324420 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:19:09.324779 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:09.316581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.28.152.189:34317)
0926 02:19:09.316804 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:09.316812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:09.318735 (+ 1923us) server_negotiation.cc:407] Connection header received
0926 02:19:09.318998 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:09.319003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:09.319082 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:09.319171 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:09.319760 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.320830 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.322820 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.323494 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.323601 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:09.324098 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:09.324228 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:09.324312 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:09.324487 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 02:19:09.324565 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:19:09.326963 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:09.316519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.28.152.190:45119)
0926 02:19:09.317220 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:09.317226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:09.318474 (+ 1248us) server_negotiation.cc:407] Connection header received
0926 02:19:09.318799 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:09.318805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:09.319014 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:09.319177 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:09.320053 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.321123 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.324260 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:09.324976 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:09.325086 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:09.325873 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:09.326344 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:09.326452 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:09.326562 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 02:19:09.326652 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":420,"threads_started":1}
I20260926 02:19:09.332224 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 19298
I20260926 02:19:09.333366 19360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:09.444295 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 19298
W20260926 02:19:09.461668 19389 connection.cc:535] server connection from 127.0.0.1:42942 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 02:19:09.462270 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 19372
I20260926 02:19:09.463464 19434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:09.570190 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 19372
I20260926 02:19:09.586970 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 19446
I20260926 02:19:09.588321 19508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:09.694842 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 19446
2026-09-26T02:19:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4319 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:19:09 dist-test-slave-hllj krb5kdc[19693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:19:09 dist-test-slave-hllj krb5kdc[19693](info): set up 2 sockets
Sep 26 02:19:09 dist-test-slave-hllj krb5kdc[19693](info): commencing operation
krb5kdc: starting...
W20260926 02:19:10.726961 29282 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/test-user.keytab.
Sep 26 02:19:10 dist-test-slave-hllj krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:19:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:19:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.152.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
Entry for principal kudu/127.28.152.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab.
I20260926 02:19:11.000622 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:43977
--webserver_interface=127.28.152.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:43977,127.28.152.189:35527,127.28.152.188:36207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:11.330574 19706 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:19:11.330940 19706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:11.331035 19706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:11.331146 19706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:11.338630 19706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:11.338742 19706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:11.338825 19706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:11.338899 19706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:11.338984 19706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:11.349180 19706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:43977,127.28.152.189:35527,127.28.152.188:36207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.190.keytab
--principal=kudu/127.28.152.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.190:43977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.152.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:11.350706 19706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:11.352077 19706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:11.363909 19711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:11.363931 19714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:11.363868 19712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:11.364717 19706 server_base.cc:1034] running on GCE node
I20260926 02:19:11.365602 19706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:11.367096 19706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:11.368321 19706 hybrid_clock.cc:648] HybridClock initialized: now 1790389151368272 us; error 59 us; skew 500 ppm
I20260926 02:19:11.368659 19706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:11 dist-test-slave-hllj krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:11.371984 19706 init.cc:377] Logged in from keytab as kudu/127.28.152.190@KRBTEST.COM (short username kudu)
I20260926 02:19:11.373564 19706 webserver.cc:460] Webserver started at http://127.28.152.190:37745/ using document root <none> and password file <none>
I20260926 02:19:11.374287 19706 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:11.374441 19706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:11.374756 19706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:11.377864 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/data/instance:
uuid: "beafe490f7504f4297ba7e4de44d515c"
format_stamp: "Formatted at 2026-09-26 02:19:11 on dist-test-slave-hllj"
server_key: "74fa23ad1bc61fb2babc88f609b627d0"
server_key_iv: "da2d9fc25e11dbc914474857dd5c15bf"
server_key_version: "encryptionkey@0"
I20260926 02:19:11.378680 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/wal/instance:
uuid: "beafe490f7504f4297ba7e4de44d515c"
format_stamp: "Formatted at 2026-09-26 02:19:11 on dist-test-slave-hllj"
server_key: "74fa23ad1bc61fb2babc88f609b627d0"
server_key_iv: "da2d9fc25e11dbc914474857dd5c15bf"
server_key_version: "encryptionkey@0"
I20260926 02:19:11.384224 19706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:19:11.388121 19721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:11.389721 19706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:11.389918 19706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/wal
uuid: "beafe490f7504f4297ba7e4de44d515c"
format_stamp: "Formatted at 2026-09-26 02:19:11 on dist-test-slave-hllj"
server_key: "74fa23ad1bc61fb2babc88f609b627d0"
server_key_iv: "da2d9fc25e11dbc914474857dd5c15bf"
server_key_version: "encryptionkey@0"
I20260926 02:19:11.390147 19706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:11.421775 19706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:11.426335 19706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:11.426613 19706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:11.444782 19706 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.190:43977
I20260926 02:19:11.444804 19772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.190:43977 every 8 connection(s)
I20260926 02:19:11.446233 19706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/data/info.pb
I20260926 02:19:11.450176 19773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:11.456248 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 19706
I20260926 02:19:11.456449 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-0/wal/instance
I20260926 02:19:11.456986 29282 external_mini_cluster.cc:1428] Setting key 5ed0098731ec35989096a2dc239c0dfa
I20260926 02:19:11.457283 19773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:19:11 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.152.189@KRBTEST.COM; defaulting to no policy
I20260926 02:19:11.482321 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.461251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.28.152.190:43977)
0926 02:19:11.463419 (+ 2168us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:11.463431 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:11.463483 (+ 52us) server_negotiation.cc:407] Connection header received
0926 02:19:11.464868 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:11.464874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:11.465188 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:11.465378 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:11.466940 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:11.468227 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:11.470202 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:11.470800 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:11.470944 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:11.474757 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:11.474765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:11.474779 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:11.474808 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:11.477898 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:11.478617 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:11.478624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:11.478626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:11.478789 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:11.479165 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:11.479168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:11.479170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:11.479708 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:11.479900 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:11.480474 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 02:19:11.480762 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1945,"thread_start_us":1700,"threads_started":1}
I20260926 02:19:11.482354 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.461251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:43977 (local address 127.0.0.1:51552)
0926 02:19:11.462147 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:11.462248 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:11.464464 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:11.465424 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:11.465432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:11.466143 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:11.466623 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:11.466649 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:11.468410 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:11.468415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:11.470042 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:11.470059 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:11.471055 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:11.471066 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:11.471253 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:11.471970 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:11.471997 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:11.474536 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:11.478041 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:11.478053 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:11.478067 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:11.478474 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:11.478901 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:11.478906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:11.478909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:11.479061 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:11.479847 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:11.479856 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:11.480080 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 02:19:11.480308 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 02:19:11.480939 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":132,"thread_start_us":366,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.189@KRBTEST.COM" created.
I20260926 02:19:11.485705 19773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:19:11.486783 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.486316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48284)
0926 02:19:11.486503 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:11.486633 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 02:19:11.487082 19725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
I20260926 02:19:11.488638 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.488243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48294)
0926 02:19:11.488401 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:11.488484 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 02:19:11.488873 19773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:35527: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab.
Entry for principal kudu/127.28.152.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab.
I20260926 02:19:11.515815 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.189:35527
--webserver_interface=127.28.152.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab
--principal=kudu/127.28.152.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:43977,127.28.152.189:35527,127.28.152.188:36207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:19:11.538247 19773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } attempt: 1
I20260926 02:19:11.539464 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.539018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48310)
0926 02:19:11.539204 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:11.539325 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 02:19:11.540413 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.540068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48326)
0926 02:19:11.540212 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:11.540279 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 02:19:11.540598 19773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:35527: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
I20260926 02:19:11.608888 19773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } attempt: 2
I20260926 02:19:11.610219 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.609690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48330)
0926 02:19:11.609966 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:11.610065 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 02:19:11.611305 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.610929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48346)
0926 02:19:11.611079 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:11.611167 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:19:11.611502 19773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:35527: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
I20260926 02:19:11.702766 19773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } attempt: 3
I20260926 02:19:11.703961 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.703495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48358)
0926 02:19:11.703727 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:11.703825 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 02:19:11.705044 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.704660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48372)
0926 02:19:11.704817 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:11.704893 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 02:19:11.705281 19773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:35527: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
I20260926 02:19:11.848529 19773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } attempt: 4
I20260926 02:19:11.849912 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.849350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48388)
0926 02:19:11.849682 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:11.849777 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260926 02:19:11.850937 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:11.850585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48390)
0926 02:19:11.850731 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:11.850818 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 02:19:11.851130 19773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.189:35527: Network error: Client connection negotiation failed: client connection to 127.28.152.189:35527: connect: Connection refused (error 111)
W20260926 02:19:11.853600 19780 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:19:11.853924 19780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:11.854013 19780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:11.854120 19780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:11.861423 19780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:11.861515 19780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:11.861605 19780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:11.861668 19780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:11.861748 19780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:11.872305 19780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:43977,127.28.152.189:35527,127.28.152.188:36207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.189.keytab
--principal=kudu/127.28.152.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.189:35527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.152.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:11.873874 19780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:11.875258 19780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:11.887056 19785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:11.887140 19786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:11.887161 19788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:11.888430 19780 server_base.cc:1034] running on GCE node
I20260926 02:19:11.889262 19780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:11.890861 19780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:11.892094 19780 hybrid_clock.cc:648] HybridClock initialized: now 1790389151892022 us; error 45 us; skew 500 ppm
I20260926 02:19:11.892444 19780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:11 dist-test-slave-hllj krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:11.895854 19780 init.cc:377] Logged in from keytab as kudu/127.28.152.189@KRBTEST.COM (short username kudu)
I20260926 02:19:11.897428 19780 webserver.cc:460] Webserver started at http://127.28.152.189:37805/ using document root <none> and password file <none>
I20260926 02:19:11.898147 19780 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:11.898257 19780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:11.898581 19780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:11.901752 19780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/data/instance:
uuid: "4523a8d1891f4fbdb0cf4a72560646f0"
format_stamp: "Formatted at 2026-09-26 02:19:11 on dist-test-slave-hllj"
server_key: "18c92aaf8679f31445796c434a22c7db"
server_key_iv: "18596ac47d6ea4af2c1fe313f2c707a0"
server_key_version: "encryptionkey@0"
I20260926 02:19:11.902557 19780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/wal/instance:
uuid: "4523a8d1891f4fbdb0cf4a72560646f0"
format_stamp: "Formatted at 2026-09-26 02:19:11 on dist-test-slave-hllj"
server_key: "18c92aaf8679f31445796c434a22c7db"
server_key_iv: "18596ac47d6ea4af2c1fe313f2c707a0"
server_key_version: "encryptionkey@0"
I20260926 02:19:11.909089 19780 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 02:19:11.913026 19795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:11.914747 19780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:11.914949 19780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/wal
uuid: "4523a8d1891f4fbdb0cf4a72560646f0"
format_stamp: "Formatted at 2026-09-26 02:19:11 on dist-test-slave-hllj"
server_key: "18c92aaf8679f31445796c434a22c7db"
server_key_iv: "18596ac47d6ea4af2c1fe313f2c707a0"
server_key_version: "encryptionkey@0"
I20260926 02:19:11.915169 19780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:11.971626 19780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:11.976274 19780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:11.976542 19780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:11.994490 19780 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.189:35527
I20260926 02:19:11.994539 19846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.189:35527 every 8 connection(s)
I20260926 02:19:11.995913 19780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/data/info.pb
I20260926 02:19:11.999877 19847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:12.002537 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 19780
I20260926 02:19:12.002732 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-1/wal/instance
I20260926 02:19:12.003273 29282 external_mini_cluster.cc:1428] Setting key 32e30085ac53d93e6f5346696008edf1
I20260926 02:19:12.008770 19847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.152.188@KRBTEST.COM; defaulting to no policy
I20260926 02:19:12.032578 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.012208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.28.152.190:43977)
0926 02:19:12.012740 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.012746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.013951 (+ 1205us) server_negotiation.cc:407] Connection header received
0926 02:19:12.016322 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.016328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.016413 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.016517 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.018702 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.019596 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.021767 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.022255 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.022368 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.026499 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.026505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.026509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.026556 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.029293 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.030086 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.030090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.030093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.030227 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.030645 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.030649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.030651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.030952 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.031092 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.031961 (+ 869us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.032204 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":260,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.152.188@KRBTEST.COM" created.
I20260926 02:19:12.033667 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.012848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:43977 (local address 127.0.0.1:51566)
0926 02:19:12.013708 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.013761 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:12.015980 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:12.016789 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:12.016801 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:12.017603 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:12.018433 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.018458 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.019765 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.019771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.021597 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.021611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.022483 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.022489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.022768 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.023527 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:12.023557 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:12.026208 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:12.029441 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.029451 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.029471 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.029944 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.030332 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.030338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.030341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.030535 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.031082 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:12.031090 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:12.031513 (+ 423us) client_negotiation.cc:769] Sending connection context
0926 02:19:12.031822 (+ 309us) client_negotiation.cc:240] Negotiation successful
0926 02:19:12.032378 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":369,"threads_started":1}
I20260926 02:19:12.035671 19847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:19:12.050359 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.036571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48394)
0926 02:19:12.036777 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.036796 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:12.036894 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:12.037830 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:12.037835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:12.038258 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:12.038392 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.038401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.039775 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.039779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.040613 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.040624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.041471 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.041477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.041645 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.042255 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:12.042273 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:12.044442 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:12.047631 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.047637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.047640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.048122 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.048532 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.048536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.048539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.048710 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.049408 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:12.049413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:12.049574 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 02:19:12.049716 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 02:19:12.050011 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:19:12.050657 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.036718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.28.152.189:35527)
0926 02:19:12.037381 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.037388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.037413 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:19:12.037460 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.037465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.037658 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.037798 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.038551 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.039637 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.040738 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.041335 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.041445 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.044664 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.044676 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.044692 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.044727 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.047485 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.048274 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.048281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.048283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.048423 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.048807 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.048810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.048812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.049238 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.049446 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.049823 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.050123 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":508,"thread_start_us":339,"threads_started":1}
I20260926 02:19:12.053385 19847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:19:12.054668 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.054191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60528)
0926 02:19:12.054388 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.054510 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 02:19:12.054929 19798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
I20260926 02:19:12.056464 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.056127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60530)
0926 02:19:12.056288 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.056361 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 02:19:12.056656 19847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36207: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.152.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab.
Entry for principal kudu/127.28.152.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab.
I20260926 02:19:12.068043 29282 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
/tmp/dist-test-taskCd4s0O/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.188:36207
--webserver_interface=127.28.152.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab
--principal=kudu/127.28.152.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.152.148:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.152.190:43977,127.28.152.189:35527,127.28.152.188:36207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:19:12.105962 19847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } attempt: 1
I20260926 02:19:12.107065 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.106595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60540)
0926 02:19:12.106807 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.106910 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 02:19:12.108101 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.107778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60548)
0926 02:19:12.107917 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.107997 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 02:19:12.108340 19847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36207: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
I20260926 02:19:12.150414 19773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } attempt: 5
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:19:12.165066 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.151308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.28.152.189:35527)
0926 02:19:12.151500 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.151504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.151525 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:19:12.151739 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.151743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.151830 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.151919 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.152907 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.153657 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.155382 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.155885 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.156020 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.159627 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.159633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.159637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.159676 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.162307 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.163166 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.163171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.163173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.163360 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.163771 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.163775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.163777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.164072 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.164183 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.164657 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.164843 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:19:12.165232 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.151206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48400)
0926 02:19:12.151434 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.151457 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:12.151596 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:12.151984 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:12.151989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:12.152499 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:12.152731 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.152743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.153813 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.153818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.155190 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.155207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.156081 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.156087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.156283 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.156964 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:12.156983 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:12.159406 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:12.162470 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.162476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.162481 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.163032 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.163470 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.163475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.163479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.163670 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.164194 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:12.164199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:12.164380 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 02:19:12.164544 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 02:19:12.164873 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 02:19:12.166504 19773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:19:12.167727 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.167267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60550)
0926 02:19:12.167455 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.167560 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 02:19:12.168864 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.168472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60566)
0926 02:19:12.168626 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.168710 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:19:12.169096 19773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36207: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
I20260926 02:19:12.176546 19847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } attempt: 2
I20260926 02:19:12.177755 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.177279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60576)
0926 02:19:12.177489 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.177587 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 02:19:12.178781 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.178428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60592)
0926 02:19:12.178583 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.178660 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 02:19:12.178984 19847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36207: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
I20260926 02:19:12.220403 19773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } attempt: 1
I20260926 02:19:12.221683 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.221103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60598)
0926 02:19:12.221371 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.221483 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260926 02:19:12.222703 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.222339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60614)
0926 02:19:12.222482 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.222567 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 02:19:12.222949 19773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36207: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
I20260926 02:19:12.270244 19847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } attempt: 3
I20260926 02:19:12.271421 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.270924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60626)
0926 02:19:12.271141 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.271238 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 02:19:12.272708 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.272219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60642)
0926 02:19:12.272499 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.272585 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
W20260926 02:19:12.272994 19847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36207: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
I20260926 02:19:12.291213 19773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } attempt: 2
I20260926 02:19:12.292500 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.291933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60652)
0926 02:19:12.292176 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.292289 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 02:19:12.293716 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.293286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60668)
0926 02:19:12.293455 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.293570 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 02:19:12.293985 19773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36207: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
I20260926 02:19:12.400277 19773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } attempt: 3
I20260926 02:19:12.401530 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.401020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60672)
0926 02:19:12.401261 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.401363 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 02:19:12.402588 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.402193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60686)
0926 02:19:12.402402 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.402481 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 02:19:12.402812 19773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36207: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
W20260926 02:19:12.411608 19855 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:19:12.411963 19855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:12.412072 19855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:12.412181 19855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 02:19:12.416244 19847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } attempt: 4
I20260926 02:19:12.417616 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.416972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60700)
0926 02:19:12.417278 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.417404 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 02:19:12.418641 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.418299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60716)
0926 02:19:12.418443 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.418521 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:19:12.418921 19847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.152.188:36207: Network error: Client connection negotiation failed: client connection to 127.28.152.188:36207: connect: Connection refused (error 111)
W20260926 02:19:12.420512 19855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:12.420608 19855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:12.420697 19855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:12.420764 19855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:12.420845 19855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:12.431384 19855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.152.148:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.152.190:43977,127.28.152.189:35527,127.28.152.188:36207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/krb5kdc/kudu_127.28.152.188.keytab
--principal=kudu/127.28.152.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.152.188:36207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.152.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:12.432716 19855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:12.434118 19855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:12.446542 19863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:12.446574 19861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:12.449517 19860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:12.453299 19855 server_base.cc:1034] running on GCE node
I20260926 02:19:12.454095 19855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:12.455530 19855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:12.456774 19855 hybrid_clock.cc:648] HybridClock initialized: now 1790389152456703 us; error 44 us; skew 500 ppm
I20260926 02:19:12.457119 19855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:12.460546 19855 init.cc:377] Logged in from keytab as kudu/127.28.152.188@KRBTEST.COM (short username kudu)
I20260926 02:19:12.462218 19855 webserver.cc:460] Webserver started at http://127.28.152.188:39213/ using document root <none> and password file <none>
I20260926 02:19:12.463006 19855 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:12.463111 19855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:12.463428 19855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:12.466604 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/data/instance:
uuid: "a526446f84824c408e5f98b8c3f7a096"
format_stamp: "Formatted at 2026-09-26 02:19:12 on dist-test-slave-hllj"
server_key: "a1dce419d59101a3f2fbd6bf7380a069"
server_key_iv: "dc2dd0ca405a479770b97d2f2e116403"
server_key_version: "encryptionkey@0"
I20260926 02:19:12.467459 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/wal/instance:
uuid: "a526446f84824c408e5f98b8c3f7a096"
format_stamp: "Formatted at 2026-09-26 02:19:12 on dist-test-slave-hllj"
server_key: "a1dce419d59101a3f2fbd6bf7380a069"
server_key_iv: "dc2dd0ca405a479770b97d2f2e116403"
server_key_version: "encryptionkey@0"
I20260926 02:19:12.473179 19855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:19:12.477162 19870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:12.478838 19855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:19:12.479053 19855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/data,/tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/wal
uuid: "a526446f84824c408e5f98b8c3f7a096"
format_stamp: "Formatted at 2026-09-26 02:19:12 on dist-test-slave-hllj"
server_key: "a1dce419d59101a3f2fbd6bf7380a069"
server_key_iv: "dc2dd0ca405a479770b97d2f2e116403"
server_key_version: "encryptionkey@0"
I20260926 02:19:12.479307 19855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:12.496027 19855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:12.500474 19855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:12.500735 19855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:12.519558 19855 rpc_server.cc:307] RPC server started. Bound to: 127.28.152.188:36207
I20260926 02:19:12.519601 19921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.152.188:36207 every 8 connection(s)
I20260926 02:19:12.521092 19855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/data/info.pb
I20260926 02:19:12.523119 29282 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu as pid 19855
I20260926 02:19:12.523276 29282 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCd4s0O/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790388791636863-29282-0/minicluster-data/master-2/wal/instance
I20260926 02:19:12.523743 29282 external_mini_cluster.cc:1428] Setting key 8bf6ce33ffbb2b89d8d1fc9559aa8a43
I20260926 02:19:12.525789 19922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:12.535353 19922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
I20260926 02:19:12.548955 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.525782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.28.152.188:36207)
0926 02:19:12.526681 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.526698 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.526755 (+ 57us) server_negotiation.cc:407] Connection header received
0926 02:19:12.528970 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.529002 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.529684 (+ 682us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.530217 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.531805 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.533879 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.535305 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.536426 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.536611 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.541268 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.541276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.541287 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.541327 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.544163 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.545029 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.545036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.545042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.545314 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.545737 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.545744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.545747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.546522 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.546823 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.547301 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.547717 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":395,"threads_started":1}
I20260926 02:19:12.553093 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.538558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.190:43977 (local address 127.0.0.1:51582)
0926 02:19:12.539351 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.539394 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:12.539548 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:12.539954 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:12.539960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:12.540660 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:12.540923 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.540931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.542030 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.542034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.543135 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.543147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.543851 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.543858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.544037 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.544586 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:12.544608 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:12.547029 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:12.550140 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.550153 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.550160 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.550622 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.551013 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.551018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.551022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.551162 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.551757 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:12.551765 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:12.551981 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 02:19:12.552292 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 02:19:12.552653 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":438,"threads_started":1}
I20260926 02:19:12.553514 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.538534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.28.152.190:43977)
0926 02:19:12.539035 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.539041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.539494 (+ 453us) server_negotiation.cc:407] Connection header received
0926 02:19:12.539674 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.539680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.539769 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.539884 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.541094 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.541868 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.543274 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.543669 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.543787 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.547217 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.547222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.547225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.547274 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.550002 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.550747 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.550753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.550756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.550901 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.551270 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.551274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.551276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.551612 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.551782 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.552613 (+ 831us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.552825 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":232,"threads_started":1}
I20260926 02:19:12.554975 19922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.190@KRBTEST.COM
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:19:12.569070 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.552056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.28.152.190:43977)
0926 02:19:12.552718 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.552725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.552749 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:19:12.552798 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.552803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.552877 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.552987 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.554407 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.555363 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.557078 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.557767 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.557897 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.562951 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.562960 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.562965 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.563012 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.566324 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.567071 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.567078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.567082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.567233 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.567617 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.567621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.567625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.567917 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.568135 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.568483 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.568665 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":284,"threads_started":1}
I20260926 02:19:12.570299 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.555830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.189:35527 (local address 127.0.0.1:48410)
0926 02:19:12.556031 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.556050 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:12.556174 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:12.556462 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:12.556467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:12.556950 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:12.557191 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.557202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.558417 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.558421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.559752 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.559771 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.560677 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.560683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.560899 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.561529 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:12.561549 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:12.564227 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:12.567614 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.567621 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.567623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.568030 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.568426 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.568430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.568434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.568602 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.569353 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:12.569357 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:12.569507 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 02:19:12.569632 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 02:19:12.569941 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 02:19:12.570329 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.555842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.28.152.189:35527)
0926 02:19:12.556123 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.556128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.556144 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:19:12.556283 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.556287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.556347 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.556419 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.557336 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.558262 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.559935 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.560507 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.560613 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.564455 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.564463 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.564468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.564524 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.567469 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.568165 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.568171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.568173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.568309 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.568725 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.568730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.568733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.569098 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.569288 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.569756 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.569965 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260926 02:19:12.571370 19922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.188@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.152.189@KRBTEST.COM
I20260926 02:19:12.580031 19773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } attempt: 4
I20260926 02:19:12.587258 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.572265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60728)
0926 02:19:12.572488 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.572501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:12.572606 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:12.572900 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:12.572903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:12.573338 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:12.573484 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.573492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.574488 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.574494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.575631 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.575644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.576425 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.576431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.576598 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.577247 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:12.577267 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:12.579423 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:12.583512 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.583517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.583521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.584063 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.585550 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.585555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.585558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.585722 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.586243 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:12.586248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:12.586387 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 02:19:12.586523 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 02:19:12.586846 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260926 02:19:12.587781 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.572218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.28.152.188:36207)
0926 02:19:12.572404 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.572410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.572558 (+ 148us) server_negotiation.cc:407] Connection header received
0926 02:19:12.572698 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.572703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.572765 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.572853 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.573617 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.574360 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.575781 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.576271 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.576380 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.579596 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.579601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.579604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.579635 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.583380 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.585220 (+ 1840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.585226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.585229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.585427 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.585830 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.585835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.585838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.586135 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.586263 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.586637 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.587306 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":395,"server-negotiator.queue_time_us":65}
I20260926 02:19:12.588056 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.570900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.28.152.189:35527)
0926 02:19:12.571191 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.571196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.571214 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:19:12.571260 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.571264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.571333 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.571404 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.572813 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.573718 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.575218 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.575683 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.575813 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.580299 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.580306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.580311 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.580354 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.583693 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.584543 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.584548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.584552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.584728 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.585236 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.585240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.585242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.585587 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.585885 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.587497 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.587718 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.190@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
I20260926 02:19:12.593446 29282 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:19:12.593941 19922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096: Bootstrap starting.
I20260926 02:19:12.598361 19922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:12.596530 19748 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "beafe490f7504f4297ba7e4de44d515c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51594
I20260926 02:19:12.599725 19922 log.cc:824] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:12.601109 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.580730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60734)
0926 02:19:12.580933 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.580953 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:12.581069 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:12.582085 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:12.582090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:12.582501 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:12.582675 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.582684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.584190 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.584196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.585546 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.585563 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.586345 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.586351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.586553 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.587194 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:12.587216 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:12.594272 (+ 7056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:12.598131 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.598138 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.598142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.598736 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.599184 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.599189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.599193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.599382 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.600046 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:12.600051 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:12.600251 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 02:19:12.600400 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 02:19:12.600719 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 02:19:12.601492 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.580748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.28.152.188:36207)
0926 02:19:12.581717 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.581723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.581747 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:19:12.581792 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.581797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.581878 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.582040 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.582837 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.584064 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.585698 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.586199 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.586319 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.594535 (+ 8216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.594542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.594545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.594581 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.597962 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.598878 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.598884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.598887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.599060 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.599512 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.599517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.599520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.599921 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.600067 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.600638 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.601051 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":773,"threads_started":1}
I20260926 02:19:12.603348 19922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096: No bootstrap required, opened a new log
I20260926 02:19:12.606243 19773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c: Bootstrap starting.
I20260926 02:19:12.609622 19773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:12.610744 19773 log.cc:824] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:12.613668 19922 raft_consensus.cc:348] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } }
I20260926 02:19:12.614022 19922 raft_consensus.cc:729] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a526446f84824c408e5f98b8c3f7a096, State: Initialized, Role: FOLLOWER
I20260926 02:19:12.614326 19773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c: No bootstrap required, opened a new log
I20260926 02:19:12.614555 19922 consensus_queue.cc:260] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } }
I20260926 02:19:12.615522 19932 sys_catalog.cc:455] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } } }
I20260926 02:19:12.615934 19932 sys_catalog.cc:458] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:12.616745 19922 sys_catalog.cc:564] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:12.621098 19773 raft_consensus.cc:348] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } }
I20260926 02:19:12.621526 19773 raft_consensus.cc:729] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beafe490f7504f4297ba7e4de44d515c, State: Initialized, Role: FOLLOWER
I20260926 02:19:12.621981 19773 consensus_queue.cc:260] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } }
I20260926 02:19:12.622869 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } } }
I20260926 02:19:12.623299 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:12.624090 19773 sys_catalog.cc:564] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:12.624220 19748 raft_consensus.cc:482] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 02:19:12.624466 19748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:12.628269 19748 raft_consensus.cc:504] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } }
I20260926 02:19:12.631127 19748 leader_election.cc:288] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [CANDIDATE]: Term 1 election: Requested vote from peers 4523a8d1891f4fbdb0cf4a72560646f0 (127.28.152.189:35527), a526446f84824c408e5f98b8c3f7a096 (127.28.152.188:36207)
I20260926 02:19:12.632161 19897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "beafe490f7504f4297ba7e4de44d515c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a526446f84824c408e5f98b8c3f7a096"
I20260926 02:19:12.632750 19897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:12.636629 19897 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beafe490f7504f4297ba7e4de44d515c in term 1.
I20260926 02:19:12.637499 19724 leader_election.cc:302] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a526446f84824c408e5f98b8c3f7a096, beafe490f7504f4297ba7e4de44d515c; no voters:
I20260926 02:19:12.638119 19937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:12.641438 19955 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:19:12.649247 19937 raft_consensus.cc:686] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [term 1 LEADER]: Becoming Leader. State: Replica: beafe490f7504f4297ba7e4de44d515c, State: Running, Role: LEADER
I20260926 02:19:12.649941 19937 consensus_queue.cc:237] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } }
I20260926 02:19:12.644819 19855 master_runner.cc:186] Error getting master registration for 127.28.152.190:43977: OK, instance_id { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" instance_seqno: 1790389151439640 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:19:12.654793 19954 sys_catalog.cc:455] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [sys.catalog]: SysCatalogTable state changed. Reason: New leader beafe490f7504f4297ba7e4de44d515c. Latest consensus state: current_term: 1 leader_uuid: "beafe490f7504f4297ba7e4de44d515c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } } }
I20260926 02:19:12.655241 19855 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:19:12.655253 19954 sys_catalog.cc:458] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:12.656483 19959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:19:12.663204 19959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 02:19:12.679729 19727 tablet.cc:2369] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:19:12.688510 19706 master_runner.cc:186] Error getting master registration for 127.28.152.189:35527: OK, instance_id { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" instance_seqno: 1790389151989106 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:19:12.699239 19706 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:19:12.708313 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.660499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48428 (local address 127.28.152.189:35527)
0926 02:19:12.660715 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.660721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.675206 (+ 14485us) server_negotiation.cc:407] Connection header received
0926 02:19:12.681371 (+ 6165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.681380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.681505 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.681597 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.689357 (+ 7760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.690466 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.695016 (+ 4550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.695713 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.695848 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.700962 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.700970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.700974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.701024 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.704720 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.705804 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.705812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.705816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.706029 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.706507 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.706514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.706517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.706889 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.707124 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.707681 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.707943 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:19:12.708830 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.660783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.28.152.190:43977)
0926 02:19:12.660961 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.660968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.674629 (+ 13661us) server_negotiation.cc:407] Connection header received
0926 02:19:12.683773 (+ 9144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.683781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.683925 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.684055 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.689683 (+ 5628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.690924 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.693106 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.693951 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.694096 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.700992 (+ 6896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.700998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.701002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.701040 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.704839 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.706429 (+ 1590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.706436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.706440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.706611 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.707090 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.707095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.707098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.707459 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.707728 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.708277 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.708510 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 02:19:12.713199 19897 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [term 1 FOLLOWER]: Refusing update from remote peer beafe490f7504f4297ba7e4de44d515c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:12.714219 19937 consensus_queue.cc:1035] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:12.718132 19847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } attempt: 5
I20260926 02:19:12.723196 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.660755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.28.152.188:36207)
0926 02:19:12.660925 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.660930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.697401 (+ 36471us) server_negotiation.cc:407] Connection header received
0926 02:19:12.697715 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.697721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.697831 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.697904 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.702556 (+ 4652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.703650 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.705932 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.706589 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.706727 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.715156 (+ 8429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.715165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.715170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.715219 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.719056 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.720647 (+ 1591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.720653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.720657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.720811 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.721352 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.721358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.721362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.721749 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.721996 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.722586 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.722835 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 02:19:12 dist-test-slave-hllj krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.152.189@KRBTEST.COM for kudu/127.28.152.188@KRBTEST.COM
I20260926 02:19:12.736428 19932 sys_catalog.cc:455] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beafe490f7504f4297ba7e4de44d515c. Latest consensus state: current_term: 1 leader_uuid: "beafe490f7504f4297ba7e4de44d515c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } } }
I20260926 02:19:12.736742 19932 sys_catalog.cc:458] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:12.737416 19967 mvcc.cc:205] Tried to move back new op lower bound from 7333433969502347264 to 7333433969260716032. Current Snapshot: MvccSnapshot[applied={T|T < 7333433969502347264}]
I20260926 02:19:12.738087 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "beafe490f7504f4297ba7e4de44d515c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } } }
I20260926 02:19:12.738384 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:12.743134 19959 catalog_manager.cc:1347] Generated new cluster ID: 64d0ae817dc14bed98a6a49d28cf6165
I20260926 02:19:12.743259 19959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:19:12.744268 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.718706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.152.188:36207 (local address 127.0.0.1:60760)
0926 02:19:12.719082 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:12.719103 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:12.719214 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:12.719929 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:12.719934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:12.720451 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:12.720704 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.720718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.722472 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.722478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.723998 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:12.724014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.725677 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.725685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:12.725947 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.726701 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:12.726730 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:12.729832 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:12.737576 (+ 7744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.737583 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.737588 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.738171 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.742023 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:12.742029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:12.742034 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:12.742256 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:12.743096 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:12.743102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:12.743329 (+ 227us) client_negotiation.cc:769] Sending connection context
0926 02:19:12.743494 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 02:19:12.743852 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260926 02:19:12.745285 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.719416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.28.152.188:36207)
0926 02:19:12.719618 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.719623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.719643 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:19:12.719693 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.719698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.719772 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.719870 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.720896 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.722144 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.724496 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.725243 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.725371 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.730078 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:12.730085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:12.730090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:12.730130 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:12.737381 (+ 7251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.738334 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.741654 (+ 3320us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.741658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.741848 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:12.742405 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:12.742410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:12.742413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:12.742836 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:12.743015 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.744586 (+ 1571us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.744862 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 02:19:12.756783 19847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0: Bootstrap starting.
I20260926 02:19:12.761631 19847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:12.767727 19847 log.cc:824] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:12.770073 19932 sys_catalog.cc:455] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "beafe490f7504f4297ba7e4de44d515c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } } }
I20260926 02:19:12.770342 19932 sys_catalog.cc:458] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:12.778690 19847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0: No bootstrap required, opened a new log
I20260926 02:19:12.783303 19959 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:19:12.785799 19959 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:19:12.789574 19847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } }
I20260926 02:19:12.790047 19847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4523a8d1891f4fbdb0cf4a72560646f0, State: Initialized, Role: FOLLOWER
I20260926 02:19:12.790637 19847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } }
I20260926 02:19:12.791893 19974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } } }
I20260926 02:19:12.792291 19974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:12.794247 19847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:12.794328 19822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "beafe490f7504f4297ba7e4de44d515c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4523a8d1891f4fbdb0cf4a72560646f0"
I20260926 02:19:12.794876 19822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:12.798506 19822 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beafe490f7504f4297ba7e4de44d515c in term 1.
I20260926 02:19:12.799666 19821 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [term 1 FOLLOWER]: Refusing update from remote peer beafe490f7504f4297ba7e4de44d515c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:12.800616 19954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:12.832206 19959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c: Generated new TSK 0
I20260926 02:19:12.833665 19959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:19:12.860087 19974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beafe490f7504f4297ba7e4de44d515c. Latest consensus state: current_term: 1 leader_uuid: "beafe490f7504f4297ba7e4de44d515c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } } }
I20260926 02:19:12.860105 19983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "beafe490f7504f4297ba7e4de44d515c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } } }
I20260926 02:19:12.860489 19974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:12.860523 19983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:19:12.861965 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "beafe490f7504f4297ba7e4de44d515c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beafe490f7504f4297ba7e4de44d515c" member_type: VOTER last_known_addr { host: "127.28.152.190" port: 43977 } } peers { permanent_uuid: "4523a8d1891f4fbdb0cf4a72560646f0" member_type: VOTER last_known_addr { host: "127.28.152.189" port: 35527 } } peers { permanent_uuid: "a526446f84824c408e5f98b8c3f7a096" member_type: VOTER last_known_addr { host: "127.28.152.188" port: 36207 } } }
I20260926 02:19:12.862250 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P beafe490f7504f4297ba7e4de44d515c [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:12.884509 19990 catalog_manager.cc:1259] Loaded cluster ID: 64d0ae817dc14bed98a6a49d28cf6165
I20260926 02:19:12.884627 19990 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0: loading cluster ID for follower catalog manager: success
I20260926 02:19:12.888254 19990 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0: acquiring CA information for follower catalog manager: success
I20260926 02:19:12.890889 19990 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4523a8d1891f4fbdb0cf4a72560646f0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 02:19:12.926875 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:12.914956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.28.152.188:36207)
0926 02:19:12.915378 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.915385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.924455 (+ 9070us) server_negotiation.cc:407] Connection header received
0926 02:19:12.924830 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.924838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.924951 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.925029 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.926598 (+ 1569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60772: BlockingRecv error: recv got EOF from 127.0.0.1:60772 (error 108)
Metrics: {"server-negotiator.queue_time_us":280}
W20260926 02:19:12.929812 19928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:12.914725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.28.152.190:43977)
0926 02:19:12.915205 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.915212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.927865 (+ 12653us) server_negotiation.cc:407] Connection header received
0926 02:19:12.928024 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.928031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.928147 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.928273 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.929464 (+ 1191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51614: BlockingRecv error: recv got EOF from 127.0.0.1:51614 (error 108)
Metrics: {"server-negotiator.queue_time_us":312}
W20260926 02:19:12.932403 19852 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:12.920881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.28.152.189:35527)
0926 02:19:12.921083 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.921089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.930569 (+ 9480us) server_negotiation.cc:407] Connection header received
0926 02:19:12.930750 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.930756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.930887 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.930979 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.932138 (+ 1159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48430: BlockingRecv error: recv got EOF from 127.0.0.1:48430 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:19:12.943822 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.937332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.28.152.189:35527)
0926 02:19:12.937534 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.937541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.938253 (+ 712us) server_negotiation.cc:407] Connection header received
0926 02:19:12.938474 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.938478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.938650 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.938802 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:12.939203 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.939973 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.941397 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.941903 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.942018 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.942562 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:12.943142 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:12.943289 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.943377 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.943477 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 02:19:12.955235 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:12.950401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.28.152.188:36207)
0926 02:19:12.950606 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.950613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.953687 (+ 3074us) server_negotiation.cc:407] Connection header received
0926 02:19:12.953919 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.953923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.954026 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.954116 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.954987 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60778: BlockingRecv error: recv got EOF from 127.0.0.1:60778 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:19:12.965837 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.959477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.28.152.189:35527)
0926 02:19:12.959702 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.959713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.960242 (+ 529us) server_negotiation.cc:407] Connection header received
0926 02:19:12.960448 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.960452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.960550 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.960630 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:12.961295 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.962084 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.963928 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.964432 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.964552 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.964987 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:12.965181 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:12.965265 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.965416 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.965504 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 02:19:12.975147 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:12.973096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.28.152.188:36207)
0926 02:19:12.973346 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.973353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.973694 (+ 341us) server_negotiation.cc:407] Connection header received
0926 02:19:12.973904 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.973910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.973990 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.974076 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:12.974932 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60788: BlockingRecv error: recv got EOF from 127.0.0.1:60788 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:19:12.988004 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:12.982237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48446 (local address 127.28.152.189:35527)
0926 02:19:12.982439 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.982446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:12.982970 (+ 524us) server_negotiation.cc:407] Connection header received
0926 02:19:12.983257 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:12.983263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:12.983389 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:12.983486 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:12.984014 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.984911 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.986314 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:12.986749 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:12.986875 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:12.987258 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:12.987377 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:12.987451 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:12.987588 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 02:19:12.987674 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 02:19:13.011543 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:12.994226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.28.152.188:36207)
0926 02:19:12.994414 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:12.994419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.009833 (+ 15414us) server_negotiation.cc:407] Connection header received
0926 02:19:13.010041 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.010046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.010143 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.010233 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:13.011280 (+ 1047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60794: BlockingRecv error: recv got EOF from 127.0.0.1:60794 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:19:13.038200 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:13.032403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48460 (local address 127.28.152.189:35527)
0926 02:19:13.032680 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.032687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.033328 (+ 641us) server_negotiation.cc:407] Connection header received
0926 02:19:13.033525 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.033531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.033606 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.033682 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:13.034115 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.034898 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.036359 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.036840 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.036952 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:13.037453 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:13.037573 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:13.037650 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:13.037805 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 02:19:13.037902 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
W20260926 02:19:13.046974 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:13.044535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60806 (local address 127.28.152.188:36207)
0926 02:19:13.044836 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.044843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.045620 (+ 777us) server_negotiation.cc:407] Connection header received
0926 02:19:13.045803 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.045809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.045889 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.045973 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:13.046739 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60806: BlockingRecv error: recv got EOF from 127.0.0.1:60806 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
I20260926 02:19:13.063207 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:13.057892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48476 (local address 127.28.152.189:35527)
0926 02:19:13.058094 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.058100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.058607 (+ 507us) server_negotiation.cc:407] Connection header received
0926 02:19:13.058796 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.058800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.058870 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.058943 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:13.059369 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.060111 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.061487 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.061941 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.062054 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:13.062435 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:13.062566 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:13.062641 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:13.062779 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 02:19:13.062884 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 02:19:13.071237 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:13.069072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.28.152.188:36207)
0926 02:19:13.069365 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.069372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.069849 (+ 477us) server_negotiation.cc:407] Connection header received
0926 02:19:13.070033 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.070039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.070112 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.070195 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:13.070969 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60810: BlockingRecv error: recv got EOF from 127.0.0.1:60810 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
I20260926 02:19:13.096717 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:13.090691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.28.152.189:35527)
0926 02:19:13.090905 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.090912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.091438 (+ 526us) server_negotiation.cc:407] Connection header received
0926 02:19:13.091647 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.091651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.091730 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.091814 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:13.092260 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.093070 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.094766 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.095335 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.095448 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:13.095942 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:13.096103 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:13.096182 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:13.096342 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 02:19:13.096441 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 02:19:13.113369 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:13.111086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.28.152.188:36207)
0926 02:19:13.111325 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.111332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.111835 (+ 503us) server_negotiation.cc:407] Connection header received
0926 02:19:13.112027 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.112033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.112114 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.112197 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:13.113077 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60826: BlockingRecv error: recv got EOF from 127.0.0.1:60826 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 02:19:13.154446 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:13.148772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.28.152.189:35527)
0926 02:19:13.149007 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.149014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.149601 (+ 587us) server_negotiation.cc:407] Connection header received
0926 02:19:13.149820 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.149824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.149925 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.150004 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:13.150475 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.151215 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.152577 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.153062 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.153234 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:13.153663 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:13.153778 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:13.153868 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:13.154021 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 02:19:13.154109 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 02:19:13.162662 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:13.160207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.28.152.188:36207)
0926 02:19:13.160411 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.160418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.160938 (+ 520us) server_negotiation.cc:407] Connection header received
0926 02:19:13.161157 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.161164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.161256 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.161349 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:13.162428 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60830: BlockingRecv error: recv got EOF from 127.0.0.1:60830 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:19:13.236811 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:13.231120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.28.152.189:35527)
0926 02:19:13.231308 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.231313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.231988 (+ 675us) server_negotiation.cc:407] Connection header received
0926 02:19:13.232183 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.232187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.232257 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.232348 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:13.232780 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.233675 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.235075 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.235529 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.235692 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:13.236099 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:13.236215 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:13.236289 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:13.236448 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 02:19:13.236534 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 02:19:13.245116 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:13.242963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.28.152.188:36207)
0926 02:19:13.243156 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.243162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.243706 (+ 544us) server_negotiation.cc:407] Connection header received
0926 02:19:13.243894 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.243900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.243995 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.244079 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:13.244922 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60846: BlockingRecv error: recv got EOF from 127.0.0.1:60846 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:19:13.383468 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:13.378333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.28.152.189:35527)
0926 02:19:13.378521 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.378527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.378979 (+ 452us) server_negotiation.cc:407] Connection header received
0926 02:19:13.379178 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.379183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.379255 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.379329 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:13.379763 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.380442 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.381810 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.382237 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.382351 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:13.382754 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:13.382880 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:13.382954 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:13.383097 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 02:19:13.383183 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 02:19:13.391547 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:13.389625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.28.152.188:36207)
0926 02:19:13.389840 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.389847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.390142 (+ 295us) server_negotiation.cc:407] Connection header received
0926 02:19:13.390344 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.390350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.390430 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.390513 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:13.391311 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60858: BlockingRecv error: recv got EOF from 127.0.0.1:60858 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:19:13.646070 19955 catalog_manager.cc:1259] Loaded cluster ID: 64d0ae817dc14bed98a6a49d28cf6165
I20260926 02:19:13.646199 19955 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096: loading cluster ID for follower catalog manager: success
I20260926 02:19:13.648886 19955 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096: acquiring CA information for follower catalog manager: success
I20260926 02:19:13.651265 19955 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a526446f84824c408e5f98b8c3f7a096: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:19:13.657195 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:13.651303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.28.152.189:35527)
0926 02:19:13.651485 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.651492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.652006 (+ 514us) server_negotiation.cc:407] Connection header received
0926 02:19:13.652178 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.652183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.652274 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.652384 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:13.652853 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.653961 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.655321 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.655966 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.656097 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:13.656464 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:13.656600 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:13.656700 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:13.656823 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 02:19:13.656907 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:19:13.672492 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:13.665560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.28.152.188:36207)
0926 02:19:13.665750 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.665756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.665954 (+ 198us) server_negotiation.cc:407] Connection header received
0926 02:19:13.666178 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.666182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.666331 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.666423 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:13.666965 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.667764 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.669878 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.670390 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.670503 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:13.671193 (+ 690us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:13.671767 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:13.671906 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:13.672026 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 02:19:13.672154 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:19:13.692018 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:13.679026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60880 (local address 127.28.152.188:36207)
0926 02:19:13.679240 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.679259 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.684156 (+ 4897us) server_negotiation.cc:407] Connection header received
0926 02:19:13.684779 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.684787 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.684880 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.684944 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:13.685830 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.687096 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.689821 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.690483 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.690627 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:13.691136 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:13.691304 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:13.691377 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:13.691587 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 02:19:13.691689 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:19:13.693454 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:13.679049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.28.152.190:43977)
0926 02:19:13.682754 (+ 3705us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.682760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.684387 (+ 1627us) server_negotiation.cc:407] Connection header received
0926 02:19:13.684756 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.684763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.684981 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.685193 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:13.686107 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.687264 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.690744 (+ 3480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.691382 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.691501 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:13.692204 (+ 703us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:13.692689 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:13.692783 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:13.693006 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 02:19:13.693097 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3567,"thread_start_us":3456,"threads_started":1}
I20260926 02:19:13.697033 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:13.679144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.28.152.189:35527)
0926 02:19:13.679373 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:13.679379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:13.689932 (+ 10553us) server_negotiation.cc:407] Connection header received
0926 02:19:13.690078 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:13.690084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:13.690177 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:13.690258 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:13.691017 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.692000 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.694455 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:13.695080 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:13.695227 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:13.696308 (+ 1081us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:13.696438 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:13.696533 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:13.696683 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 02:19:13.696764 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:19:13.699668 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 19706
I20260926 02:19:13.700713 19768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:13.832746 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 19706
I20260926 02:19:13.850759 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 19780
I20260926 02:19:13.852012 19842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:13.961791 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 19780
I20260926 02:19:13.978935 29282 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu pid 19855
I20260926 02:19:13.980216 19917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:14.097657 29282 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCd4s0O/build/asan/bin/kudu with pid 19855
2026-09-26T02:19:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4403 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8722 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (362422 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST